Augmented reality guided fitting, sizing, trialing and balancing of virtual implants on the physical joint of a patient for manual and robot assisted joint replacement

ABSTRACT

Devices and methods for performing a surgical step or surgical procedure with visual guidance using an optical head mounted display are disclosed.

RELATED APPLICATIONS

This application is a continuation application of U.S. application Ser.No. 17/323,080, filed May 18, 2020, which is a continuation of U.S.application Ser. No. 17/111,643, filed Dec. 4, 2020, now U.S. Pat. No.11,013,560, which is a continuation application of Ser. No. 17/036,631,filed Sep. 29, 2020, which is a continuation application of U.S.application Ser. No. 16/668,360, filed Oct. 30, 2019, now U.S. Pat. No.10,799,296, which is a continuation application of U.S. application Ser.No. 16/524,436, filed Jul. 29, 2019, now U.S. Pat. No. 10,603,113, whichis a continuation application of U.S. application Ser. No. 16/406,392,filed May 8, 2019, now U.S. Pat. No. 10,405,927, which is a continuationapplication of U.S. application Ser. No. 15/988,455, filed May 24, 2018,now U.S. Pat. No. 10,292,768, which is a continuation of U.S.application Ser. No. 15/843,239, filed Dec. 15, 2017, now U.S. Pat. No.9,980,780, which is a continuation of U.S. application Ser. No.15/456,084, filed Mar. 10, 2017, now U.S. Pat. No. 9,861,446, whichclaims the benefit of and priority to U.S. Provisional Application Ser.No. 62/307,476, filed Mar. 12, 2016, U.S. Provisional Application Ser.No. 62/318,157, filed Apr. 4, 2016, U.S. Provisional Application Ser.No. 62/323,716, filed Apr. 17, 2016, U.S. Provisional Application Ser.No. 62/331,995, filed May 5, 2016, U.S. Provisional Application Ser. No.62/354,780, filed Jun. 26, 2016, U.S. Provisional Application Ser. No.62/378,242, filed Aug. 23, 2016, U.S. Provisional Application Ser. No.62/393,054, filed Sep. 11, 2016, U.S. Provisional Application Ser. No.62/406,379, filed Oct. 10, 2016, U.S. Provisional Application Ser. No.62/425,019, filed Nov. 21, 2016, U.S. Provisional Application Ser. No.62/445,691, filed Jan. 12, 2017, and U.S. Provisional Application Ser.No. 62/453,484, filed Feb. 1, 2017, the entire contents of each of whichare hereby incorporated by reference in their entireties.

TECHNICAL FIELD

Aspects of the invention relate to devices and methods for performing asurgical step or surgical procedure with visual guidance using anoptical head mounted display.

BACKGROUND

With computer assisted surgery, e.g. surgical navigation or robotics,pre-operative imaging studies of the patient can be used. The imagingstudies can be displayed in the OR on an external computer monitor andthe patient's anatomy, e.g. landmarks, can be registered in relationshipto the information displayed on the monitor. Since the surgical field isin a different location and has a different view coordinate system forthe surgeon's eyes than the external computer monitor, hand-eyecoordination can be challenging for the surgeon.

SUMMARY OF THE INVENTION

Aspects of the invention provides, among other things, for asimultaneous visualization of live data of the patient, e.g. a patient'sspine or joint, and digital representations of virtual data such asvirtual cuts and/or virtual surgical guides including cut blocks ordrilling guides through an optical head mounted display (OHMD). In someembodiments, the surgical site including live data of the patient, theOHMD, and the virtual data are registered in a common coordinate system.In some embodiments, the virtual data are superimposed onto and alignedwith the live data of the patient. Unlike virtual reality head systemsthat blend out live data, the OHMD allows the surgeon to see the livedata of the patient, e.g. the surgical field, while at the same timeobserving virtual data of the patient and/or virtual surgicalinstruments or implants with a predetermined position and/or orientationusing the display of the OHMD unit.

Aspects of the invention describe novel devices for performing asurgical step or surgical procedure with visual guidance using anoptical head mounted display, e.g. by displaying virtual representationsof one or more of a virtual surgical tool, virtual surgical instrumentincluding a virtual surgical guide or cut block, virtual trial implant,virtual implant component, virtual implant or virtual device, apredetermined start point, predetermined start position, predeterminedstart orientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration.

Aspects of the invention relate to a device comprising at least oneoptical head mounted display, the device being configured to generate avirtual surgical guide. In some embodiments, the virtual surgical guideis a three-dimensional representation in digital format whichcorresponds to at least one of a portion of a physical surgical guide, aplacement indicator of a physical surgical guide, or a combinationthereof. In some embodiments, the at least one optical head mounteddisplay is configured to display the virtual surgical guide superimposedonto a physical joint based at least in part on coordinates of apredetermined position of the virtual surgical guide, and the virtualsurgical guide is configured to align the physical surgical guide or aphysical saw blade with the virtual surgical guide to guide a bone cutof the joint.

In some embodiments, the device comprises one, two, three or moreoptical head mounted displays.

In some embodiments, the virtual surgical guide is configured to guide abone cut in a knee replacement, hip replacement, shoulder jointreplacement or ankle joint replacement.

In some embodiments, the virtual surgical guide includes a virtual slotfor a virtual or a physical saw blade.

In some embodiments, the virtual surgical guide includes a planar areafor aligning a virtual or a physical saw blade.

In some embodiments, the virtual surgical guide includes two or morevirtual guide holes or paths for aligning two or more physical drills orpins.

In some embodiments, the predetermined position of the virtual surgicalguide includes anatomical information, and/or alignment information ofthe joint. For example, the anatomic and/or alignment information of thejoint can be based on at least one of coordinates of the joint, ananatomical axis of the joint, a biomechanical axis of the joint, amechanical axis, or combinations thereof.

In some embodiments, the at least one optical head mounted display isconfigured to align the virtual surgical guide based on a predeterminedlimb alignment. For example, the predetermined limb alignment can be anormal mechanical axis alignment of a leg.

In some embodiments, the at least one optical head mounted display isconfigured to align the virtual surgical guide based on a predeterminedfemoral or tibial component rotation. In some embodiments, the at leastone optical head mounted display is configured to align the virtualsurgical guide based on a predetermined flexion of a femoral componentor a predetermined slope of a tibial component.

In some embodiments, the virtual surgical guide is configured to guide aproximal femoral bone cut based on a predetermined leg length.

In some embodiments, the virtual surgical guide is configured to guide abone cut of a distal tibia or a talus in an ankle joint replacement andthe at least one optical head mounted display is configured to align thevirtual surgical guide based on a predetermined ankle alignment, whereinthe predetermined ankle alignment includes a coronal plane implantcomponent alignment, a sagittal plane implant component alignment, anaxial plane component alignment, an implant component rotation orcombinations thereof.

In some embodiments, the virtual surgical guide is configured to guide abone cut of a proximal humerus in a shoulder joint replacement and theat least one optical head mounted display is configured to align thevirtual surgical guide based on a predetermined humeral implantcomponent alignment, wherein the humeral implant component alignmentincludes a coronal plane implant component alignment, a sagittal planeimplant component alignment, an axial plane component alignment, animplant component, or combinations thereof.

In some embodiments, the predetermined position of the surgical guide isbased on a pre-operative or intra-operative imaging study, one or moreintra-operative measurements, intra-operative data or combinationsthereof.

Aspects of the invention relate to a device comprising two or moreoptical head mounted displays for two or more users, wherein the deviceis configured to generate a virtual surgical guide, wherein the virtualsurgical guide is a three-dimensional representation in digital formatwhich corresponds to at least one of a portion of a physical surgicalguide, a placement indicator of a physical surgical guide, or acombination thereof, wherein the optical head mounted display isconfigured to display the virtual surgical guide superimposed onto aphysical joint based at least in part on coordinates of a predeterminedposition of the virtual surgical guide, and wherein the virtual surgicalguide is configured for aligning the physical surgical guide or a sawblade to guide a bone cut of the joint.

Aspects of the invention relate to a device comprising at least oneoptical head mounted display and a virtual bone cut plane, wherein thevirtual bone cut plane is configured to guide a bone cut of a joint,wherein the virtual bone cut plane corresponds to at least one portionof a bone cut plane, and wherein the optical head mounted display isconfigured to display the virtual bone cut plane superimposed onto aphysical joint based at least in part on coordinates of a predeterminedposition of the virtual bone cut plane. In some embodiments, the virtualbone cut plane is configured to guide a bone cut in a predeterminedvarus or valgus orientation or in a predetermined tibial slope or in apredetermined femoral flexion of an implant component or in apredetermined leg length.

Aspects of the invention relates to a method of preparing a joint for aprosthesis in a patient. In some embodiments, the method comprisesregistering one or more optical head mounted displays worn by a surgeonor surgical assistant in a coordinate system, obtaining one or moreintra-operative measurements from the patient's physical joint todetermine one or more intra-operative coordinates, registering the oneor more intra-operative coordinates from the patient's physical joint inthe coordinate system, generating a virtual surgical guide, determininga predetermined position and/or orientation of the virtual surgicalguide based on the one or more intra-operative measurements, displayingand superimposing the virtual surgical guide, using the one or moreoptical head mounted displays, onto the physical joint based at least inpart on coordinates of the predetermined position of the virtualsurgical guide, and aligning the physical surgical guide or a physicalsaw blade with the virtual surgical guide to guide a bone cut of thejoint.

In some embodiments, the one or more optical head mounted displays areregistered in a common coordinate system. In some embodiments, thecommon coordinate system is a shared coordinate system.

In some embodiments, the virtual surgical guide is used to guide a bonecut in a knee replacement, hip replacement, shoulder joint replacementor ankle joint replacement.

In some embodiments, the predetermined position of the virtual surgicalguide determines a tibial slope for implantation of one or more tibialimplant components in a knee replacement. In some embodiments, thepredetermined position of the virtual surgical guide determines an angleof varus or valgus correction for a femoral and/or a tibial component ina knee replacement.

In some embodiments, the virtual surgical guide corresponds to aphysical distal femoral guide or cut block and the predeterminedposition of the virtual surgical guide determines a femoral componentflexion.

In some embodiments, the virtual surgical guide corresponds to aphysical anterior or posterior femoral surgical guide or cut block andthe predetermined position of the virtual surgical guide determines afemoral component rotation.

In some embodiments, the virtual surgical guide corresponds to aphysical chamfer femoral guide or cut block.

In some embodiments, the virtual surgical guide corresponds to aphysical multi-cut femoral guide or cut block and the predeterminedposition of the virtual surgical guide determines one or more of ananterior cut, posterior cut, chamfer cuts and a femoral componentrotation.

In some embodiments, the virtual surgical guide is used in a hipreplacement and the predetermined position of the virtual surgical guidedetermines a leg length after implantation. In some embodiments, thevirtual surgical guide is a virtual plane for aligning the physical sawblade to guide the bone cut of the joint.

In some embodiments, the one or more intraoperative measurements includedetecting one or more optical markers attached to the patient's joint,the operating room table, fixed structures in the operating room orcombinations thereof. In some embodiments, one or more cameras or imagecapture or video capture systems included in the optical head mounteddisplay detect one or more optical markers including their coordinates(x, y, z) and at least one or more of a position, orientation,alignment, direction of movement or speed of movement of the one or moreoptical markers.

In some embodiments, registration of one or more of optical head mounteddisplays, surgical site, joint, spine, surgical instruments or implantcomponents can be performed with use of spatial mapping techniques.

In some embodiments, registration of one or more of optical head mounteddisplays, surgical site, joint, spine, surgical instruments or implantcomponents can be performed with use of depth sensors.

In some embodiments, the virtual surgical guide is used to guide a bonecut of a distal tibia or a talus in an ankle joint replacement and theone or more optical head mounted display is used to align the virtualsurgical guide based on a predetermined tibial or talar implantcomponent alignment, wherein the predetermined tibial or talar implantcomponent alignment includes a coronal plane implant componentalignment, a sagittal plane implant component alignment, an axial planecomponent alignment, an implant component rotation of an implantcomponent or combinations thereof.

In some embodiments, the virtual surgical guide is used to guide a bonecut of a proximal humerus in a shoulder joint replacement and whereinthe one or more optical head mounted display is used to align thevirtual surgical guide based on a predetermined humeral implantcomponent alignment, wherein the humeral implant component alignmentincludes a coronal plane implant component alignment, a sagittal planeimplant component alignment, an axial plane component alignment, ahumeral implant component rotation, or combinations thereof. Aspects ofthe invention relate to a system comprising at least one optical headmounted display and a virtual library of implants, wherein the virtuallibrary of implants comprises at least one virtual implant component,wherein the virtual implant component has at least one dimension thatcorresponds to a dimension of the implant component or has a dimensionthat is substantially identical to the dimension of the implantcomponent, wherein the at least one optical head mounted display isconfigured to display the virtual implant component in substantialalignment with a tissue intended for placement of the implant component,wherein the placement of the virtual implant component is intended toachieve a predetermined implant component position and/or orientation.

BRIEF DESCRIPTION OF THE DRAWINGS

Illustrative, non-limiting example embodiments will be more clearlyunderstood from the following detailed description taken in conjunctionwith the accompanying drawings.

FIG. 1 shows the use of multiple OHMDs for multiple viewer's, e.g. aprimary surgeon, second surgeon, surgical assistant(s) and/or nurses(s)according to some embodiments of the present disclosure.

FIG. 2 shows a workflow for segmentation and select subsequent stepsaccording to some embodiments of the present disclosure.

FIG. 3 illustrates an example of registering a digital hologram for aninitial surgical step, performing the surgical step and re-registeringone or more digital holograms for subsequent surgical steps according tosome embodiments of the present disclosure.

FIGS. 4A, 4B and 4C are illustrative examples of arbitrary virtualplanes in the hip and a femoral neck cut plane according to someembodiments of the present disclosure.

FIG. 5 is an illustrative example of an arbitrary virtual plane in theknee extending through the medial and lateral joint space according tosome embodiments of the present disclosure.

FIG. 6 is an illustrative flow chart that shows different methods ofaddressing inaccuracies between the changes induced by a surgical stepand the intended, projected or predetermined changes in the virtual dataof the patient according to some embodiments of the present disclosure.

FIGS. 7A-7H depict illustrative examples of a femoral neck cut andtechniques to correct a femoral neck cut according to some embodimentsof the present disclosure.

FIGS. 8A-8H depict illustrative examples of a distal femoral cut andtechniques to correct a distal femoral cut according to some embodimentsof the present disclosure.

FIGS. 9A-9G depict illustrative examples of a distal femoral cut andtechniques to correct a distal femoral cut according to some embodimentsof the present disclosure.

FIGS. 10A-10G depict illustrative examples of a distal femoral cut andproximal tibial cut and techniques to correct the cuts according to someembodiments of the present disclosure.

FIG. 11 is an illustrative example how a virtual surgical plan can begenerated using intraoperative data, e.g. intra-operative measurements,for example measurements obtained with one or more cameras, an imagecapture system or a video capture system integrated into, attached to orseparate from an optical head mount display according to someembodiments of the present disclosure.

FIG. 12 is an exemplary workflow for generating a virtual surgical planaccording to some embodiments of the present disclosure.

FIG. 13 shows an example how a virtual surgical plan can be modifiedusing intraoperative data, e.g. intraoperative measurements according tosome embodiments of the present disclosure.

FIG. 14 shows an illustrative example how multiple OHMDs can be usedduring a surgery, for example by a first surgeon, a second surgeon, asurgical assistant and/or one or more nurses and how a surgical plan canbe modified and displayed during the procedure by multiple OHMDs whilepreserving the correct perspective view of virtual data andcorresponding live data for each individual operator according to someembodiments of the present disclosure.

FIG. 15 is an example how 2D to 3D morphed data can be used or applied.

FIGS. 16A and 16B are flow charts summarizing model generation,registration and view projection for one or more OHMDs, e.g. by aprimary surgeon, second surgeon, surgical assistant nurse, or othersaccording to some embodiments of the present disclosure.

FIGS. 17A-17D are illustrative flow charts of select options andapproaches for performing spine surgery in a mixed reality environmentaccording to some embodiments of the present disclosure.

FIGS. 18A-18F are illustrative examples of displaying a virtualacetabular reaming axis using one or more OHMDs and aligning a physicalacetabular reamer with the virtual reaming axis for placing anacetabular cup with a predetermined cup angle, offset, medial or lateralposition and/or anteversion according to some embodiments of the presentdisclosure.

FIGS. 19A-19D provide an illustrative, non-limiting example of the useof virtual surgical guides such as a distal femoral cut block displayedby an OHMD and physical surgical guides such as physical distal femoralcut blocks for knee replacement according to some embodiments of thepresent disclosure.

FIGS. 20A-20C provide an illustrative, non-limiting example of the useof virtual surgical guides such as an AP femoral cut block displayed byan OHMD and physical surgical guides such as physical AP cut blocks forknee replacement according to some embodiments of the presentdisclosure.

FIGS. 21A-21F provide an illustrative, non-limiting example of the useof virtual surgical guides such as a virtual proximal tibial cut guidedisplayed by an OHMD and physical surgical guides such as physicalproximal tibial cut guide according to some embodiments of the presentdisclosure.

FIGS. 22A and 22B show AP and lateral views demonstrating exemplarynormal ACL including antero-medial and postero-lateral fibers.

FIGS. 22C and 22D show AP and lateral views demonstrating exemplary ACLtunnels (solid straight lines) on femoral side and tibial side.

FIGS. 22E and 22F show AP and lateral views demonstrating exemplaryvirtual ACL tunnels on femoral side and tibial side (straight brokenlines) according to some embodiments of the present disclosure.

FIGS. 22G and 22H show AP and lateral views demonstrating exemplaryvirtual ACL graft on femoral side and tibial side extending throughintra-articular space between femur and tibia (straight solid lines)according to some embodiments of the present disclosure.

FIG. 23 is an illustrative non-limiting flow chart describing differentapproaches to planning the location, position, orientation, alignmentand/or direction of one or more femoral or tibial tunnels (e.g. forsingle or double bundle technique) or for placing an ACL graft accordingto some embodiments of the present disclosure.

DETAILED DESCRIPTION OF THE INVENTION

Various exemplary embodiments will be described more fully hereinafterwith reference to the accompanying drawings, in which some exampleembodiments are shown. The present inventive concept may, however, beembodied in many different forms and should not be construed as limitedto the example embodiments set forth herein. Rather, these exampleembodiments are provided so that this disclosure will be thorough andcomplete, and will fully convey the scope of the present inventiveconcept to those skilled in the art. In the drawings, the sizes andrelative sizes of layers and regions may be exaggerated for clarity.Like numerals refer to like elements throughout.

The term live data of the patient, as used herein, includes the surgicalsite, anatomy, anatomic structures or tissues and/or pathology,pathologic structures or tissues of the patient as seen by the surgeon'sor viewer's eyes without information from virtual data, stereoscopicviews of virtual data, or imaging studies. The term live data of thepatient does not include internal or subsurface tissues or structures orhidden tissues or structures that can only be seen with assistance of acomputer monitor or OHMD.

The terms real surgical instrument, actual surgical instrument, physicalsurgical instrument and surgical instrument are used interchangeablythroughout the application; the terms real surgical instrument, actualsurgical instrument, physical surgical instrument and surgicalinstrument do not include virtual surgical instruments. For example, thephysical surgical instruments can be surgical instruments provided bymanufacturers or vendors for spinal surgery, pedicle screwinstrumentation, anterior spinal fusion, knee replacement, hipreplacement, ankle replacement and/or shoulder replacement; physicalsurgical instruments can be, for example, cut blocks, pin guides, awls,reamers, impactors, broaches. Physical surgical instruments can bere-useable or disposable or combinations thereof. Physical surgicalinstruments can be patient specific. The term virtual surgicalinstrument does not include real surgical instrument, actual surgicalinstrument, physical surgical instrument and surgical instrument.

The terms real surgical tool, actual surgical tool, physical surgicaltool and surgical tool are used interchangeably throughout theapplication; the terms real surgical tool, actual surgical tool,physical surgical tool and surgical tool do not include virtual surgicaltools. The physical surgical tools can be surgical tools provided bymanufacturers or vendors. For example, the physical surgical tools canbe pins, drills, saw blades, retractors, frames for tissue distractionand other tools used for orthopedic, neurologic, urologic orcardiovascular surgery. The term virtual surgical tool does not includereal surgical tool, actual surgical tool, physical surgical tool andsurgical tool.

The terms real implant or implant component, actual implant or implantcomponent, physical implant or implant component and implant or implantcomponent are used interchangeably throughout the application; the termsreal implant or implant component, actual implant or implant component,physical implant or implant component and implant or implant componentdo not include virtual implant or implant components. The physicalimplants or implant components can be implants or implant componentsprovided by manufacturers or vendors. For example, the physical surgicalimplants can be a pedicle screw, a spinal rod, a spinal cage, a femoralor tibial component in a knee replacement, an acetabular cup or afemoral stem and head in hip replacement. The term virtual implant orimplant component does not include real implant or implant component,actual implant or implant component, physical implant or implantcomponent and implant or implant component.

With surgical navigation, a first virtual instrument can be displayed ona computer monitor which is a representation of a physical instrumenttracked with navigation markers, e.g. infrared or RF markers, and theposition and/or orientation of the first virtual instrument can becompared with the position and/or orientation of a corresponding secondvirtual instrument generated in a virtual surgical plan. Thus, withsurgical navigation the positions and/or orientations of the first andthe second virtual instruments are compared.

Aspects of the invention relates to devices, systems and methods forpositioning a virtual path, virtual plane, virtual tool, virtualsurgical instrument or virtual implant component in a mixed realityenvironment using a head mounted display device, optionally coupled toone or more processing units.

With guidance in mixed reality environment, a virtual surgical guide,tool, instrument or implant can be superimposed onto the physical joint,spine or surgical site. Further, the physical guide, tool, instrument orimplant can be aligned with the virtual surgical guide, tool, instrumentor implant displayed or projected by the OHMD. Thus, guidance in mixedreality environment does not need to use a plurality of virtualrepresentations of the guide, tool, instrument or implant and does notneed to compare the positions and/or orientations of the plurality ofvirtual representations of the virtual guide, tool, instrument orimplant.

In various embodiments, the OHMD can display one or more of a virtualsurgical tool, virtual surgical instrument including a virtual surgicalguide or virtual cut block, virtual trial implant, virtual implantcomponent, virtual implant or virtual device, predetermined start point,predetermined start position, predetermined start orientation oralignment, predetermined intermediate point(s), predeterminedintermediate position(s), predetermined intermediate orientation oralignment, predetermined end point, predetermined end position,predetermined end orientation or alignment, predetermined path,predetermined plane, predetermined cut plane, predetermined contour oroutline or cross-section or surface features or shape or projection,predetermined depth marker or depth gauge, predetermined angle ororientation or rotation marker, predetermined axis, e.g. rotation axis,flexion axis, extension axis, predetermined axis of the virtual surgicaltool, virtual surgical instrument including virtual surgical guide orcut block, virtual trial implant, virtual implant component, implant ordevice, estimated or predetermined non-visualized portions for one ormore devices or implants or implant components or surgical instrumentsor surgical tools, and/or one or more of a predetermined tissue changeor alteration.

Any of a position, location, orientation, alignment, direction, speed ofmovement, force applied of a surgical instrument or tool, virtual and/orphysical, can be predetermined using, for example, pre-operative imagingstudies, pre-operative data, pre-operative measurements, intra-operativeimaging studies, intra-operative data, and/or intra-operativemeasurements.

Any of a position, location, orientation, alignment, sagittal planealignment, coronal plane alignment, axial plane alignment, rotation,slope of implantation, angle of implantation, flexion of implantcomponent, offset, anteversion, retroversion, and position, location,orientation, alignment relative to one or more anatomic landmarks,position, location, orientation, alignment relative to one or moreanatomic planes, position, location, orientation, alignment relative toone or more anatomic axes, position, location, orientation, alignmentrelative to one or more biomechanical axes, position, location,orientation, alignment relative to a mechanical axis of a trial implant,an implant component or implant, virtual and/or physical, can bepredetermined using, for example, pre-operative imaging studies,pre-operative data, pre-operative measurements, intra-operative imagingstudies, intra-operative data, and/or intra-operative measurements.Intra-operative measurements can include measurements for purposes ofregistration, e.g. of a joint, a spine, a surgical site, a bone, acartilage, an OHMD, a surgical tool or instrument, a trial implant, animplant component or an implant.

In some embodiments, multiple coordinate systems can be used instead ofa common or shared coordinate system. In this case, coordinate transferscan be applied from one coordinate system to another coordinate system,for example for registering the OHMD, live data of the patient includingthe surgical site, virtual instruments and/or virtual implants andphysical instruments and physical implants.

Optical Head Mounted Displays

In some embodiments of the invention, a pair of glasses is utilized. Theglasses can include an optical head-mounted display. An opticalhead-mounted display (OHMD) can be a wearable display that has thecapability of reflecting projected images as well as allowing the userto see through it. Various types of OHMDs can be used in order topractice the invention. These include curved mirror or curved combinerOHMDs as well as wave-guide or light-guide OHMDs. The OHMDs canoptionally utilize diffraction optics, holographic optics, polarizedoptics, and reflective optics.

Traditional input devices that can be used with the OHMDs include, butare not limited to touchpad or buttons, smartphone controllers, speechrecognition, and gesture recognition. Advanced interfaces are possible,e.g. a brain-computer interface.

Optionally, a computer or server or a workstation can transmit data tothe OHMD. The data transmission can occur via cable, Bluetooth, WiFi,optical signals and any other method or mode of data transmission knownin the art. The OHMD can display virtual data, e.g. virtual data of thepatient, in uncompressed form or in compressed form. Virtual data of apatient can optionally be reduced in resolution when transmitted to theOHMD or when displayed by the OHMD.

When virtual data are transmitted to the OHMD, they can be in compressedform during the transmission. The OHMD can then optionally decompressthem so that uncompressed virtual data are being displayed by the OHMD.

Alternatively, when virtual data are transmitted to the OHMD, they canbe of reduced resolution during the transmission, for example byincreasing the slice thickness of image data prior to the transmission.The OHMD can then optionally increase the resolution, for example byre-interpolating to the original slice thickness of the image data oreven thinner slices so that virtual data with resolution equal to orgreater than the original virtual data or at least greater in resolutionthan the transmitted data are being displayed by the OHMD.

In some embodiments, the OHMD can transmit data back to a computer, aserver or a workstation. Such data can include, but are not limited to:

-   -   Positional, orientational or directional information about the        OHMD or the operator or surgeon wearing the OHMD    -   Changes in position, orientation or direction of the OHMD    -   Data generated by one or more IMUs    -   Data generated by markers (radiofrequency, optical, light,        other) attached to, integrated with or coupled to the OHMD    -   Data generated by a surgical navigation system attached to,        integrated with or coupled to the OHMD    -   Data generated by an image and/or video capture system attached        to, integrated with or coupled to the OHMD    -   Parallax data, e.g. using two or more image and/or video capture        systems attached to, integrated with or coupled to the OHMD, for        example one positioned over or under or near the left eye and a        second positioned over or under or near the right eye    -   Distance data, e.g. parallax data generated by two or more image        and/or video capture systems evaluating changes in distance        between the OHMD and a surgical field or an object    -   Motion parallax data    -   Data related to calibration or registration phantoms (see other        sections of this specification)    -   Any type of live data of the patient captured by the OHMD        including image and/or video capture systems attached to,        integrated with or coupled to the OHMD        -   For example, alterations to a live surgical site        -   For example, use of certain surgical instruments detected by            the image and/or video capture system        -   For example, use of certain medical devices or trial            implants detected by the image and/or video capture system    -   Any type of modification to a surgical plan        -   Portions or aspects of a live surgical plan        -   Portions or aspects of a virtual surgical plan

Radiofrequency tags used throughout the embodiments can be of active orpassive kind with or without a battery.

Exemplary optical head mounted displays include the ODG R-7, R-8 and R-8smart glasses from ODG (Osterhout Group, San Francisco, Calif.), theNVIDIA 942 3-D vision wireless glasses (NVIDIA, Santa Clara, Calif.) andthe Microsoft HoloLens (Microsoft, Redmond, Wis.).

The Microsoft HoloLens is manufactured by Microsoft. It is a pair ofaugmented reality smart glasses. Hololens can use the Windows 10operating system. The front portion of the Hololens includes, amongothers, sensors, related hardware, several cameras and processors. Thevisor includes a pair of transparent combiner lenses, in which theprojected images are displayed. The HoloLens can be adjusted for theinterpupillary distance (IPD) using an integrated program thatrecognizes gestures. A pair of speakers is also integrated. The speakersdo not exclude external sounds and allow the user to hear virtualsounds. A USB 2.0 micro-B receptacle is integrated. A 3.5 mm audio jackis also present.

The HoloLens has an inertial measurement unit (IMU) with anaccelerometer, gyroscope, and a magnetometer, four environment mappingsensors/cameras (two on each side), a depth camera with a 120°×120°angle of view, a 2.4-megapixel photographic video camera, afour-microphone array, and an ambient light sensor.

Hololens has an Intel Cherry Trail SoC containing the CPU and GPU.HoloLens includes also a custom-made Microsoft Holographic ProcessingUnit (HPU). The SoC and the HPU each have 1 GB LPDDR3 and share 8 MBSRAM, with the SoC also controlling 64 GB eMMC and running the Windows10 operating system. The HPU processes and integrates data from thesensors, as well as handling tasks such as spatial mapping, gesturerecognition, and voice and speech recognition. HoloLens includes a IEEE802.11ac Wi-Fi and Bluetooth 4.1 Low Energy (LE) wireless connectivity.The headset uses Bluetooth LE and can connect to a Clicker, afinger-operating input device that can be used for selecting menus andfunctions.

A number of applications are available for Microsoft Hololens, forexample a catalogue of holograms, HoloStudio, a 3D modelling applicationby Microsoft with 3D print capability, Autodesk Maya 3D creationapplication FreeForm, integrating HoloLens with the Autodesk Fusion 360cloud-based 3D development application, and others.

HoloLens utilizing the HPU can employ sensual and natural interfacecommands—voice, gesture, and gesture. Gaze commands, e.g. head-tracking,allows the user to bring application focus to whatever the user isperceiving. Any virtual application or button can be are selected usingan air tap method, similar to clicking a virtual computer mouse. The tapcan be held for a drag simulation to move an display. Voice commands canalso be utilized.

The HoloLens shell utilizes many components or concepts from the Windowsdesktop environment. A bloom gesture for opening the main menu isperformed by opening one's hand, with the palm facing up and the fingersspread. Windows can be dragged to a particular position, locked and/orresized. Virtual windows or menus can be fixed at locations or physicalobjects. Virtual windows or menus can move with the user or can be fixedin relationship to the user. Or they can follow the user as he or shemoves around.

The Microsoft HoloLens App for Windows 10 PC's and Windows 10 Mobiledevices can be used by developers to run apps and to view live streamfrom the HoloLens user's point of view, and to capture augmented realityphotos and videos.

Almost all Universal Windows Platform apps can run on Hololens. Theseapps can be projected in 2D. Select Windows 10 APIs are currentlysupported by HoloLens. Hololens apps can also be developed on Windows 10PC's. Holographic applications can use Windows Holographic APIs. Unityand Vuforia are some apps that can be utilized. Applications can also bedeveloped using DirectX and Windows API's.

Computer Graphics Viewing Pipeline

In some embodiments of the invention, the optical head mount displayuses a computer graphics viewing pipeline that consists of the followingsteps to display 3D objects or 2D objects positioned in 3D space orother computer generated objects and models FIG. 16B:

-   -   1. Registration    -   2. View projection

Registration:

The different objects to be displayed by the OHMD computer graphicssystem (for instance virtual anatomical models, virtual models ofinstruments, geometric and surgical references and guides) are initiallyall defined in their own independent model coordinate system. During theregistration process, spatial relationships between the differentobjects are defined, and each object is transformed from its own modelcoordinate system into a common global coordinate system. Differenttechniques that are described below can be applied for the registrationprocess.

For augmented reality OHMDs that superimpose computer-generated objectswith live views of the physical environment, the global coordinatesystem is defined by the environment. A process called spatial mapping,described below, creates a computer representation of the environmentthat allows for merging and registration with the computer-generatedobjects, thus defining a spatial relationship between thecomputer-generated objects and the physical environment.

View Projection:

Once all objects to be displayed have been registered and transformedinto the common global coordinate system, they are prepared for viewingon a display by transforming their coordinates from the globalcoordinate system into the view coordinate system and subsequentlyprojecting them onto the display plane. This view projection step usesthe viewpoint and view direction to define the transformations appliedin this step. For stereoscopic displays, such as an OHMD, two differentview projections can be used, one for the left eye and the other one forthe right eye. For augmented reality OHMDs the position of the viewpointand view direction relative to the physical environment can be known inorder to correctly superimpose the computer-generated objects with thephysical environment. As the viewpoint and view direction change, forexample due to head movement, the view projections are updated so thatthe computer-generated display follows the new view.

Eye Tracking Systems

The present invention provides for methods of using the human eyeincluding eye movements and lid movements as well as movements inducedby the peri-orbital muscles for executing computer commands. Theinvention provides also for methods of executing computer commands byway of facial movements and movements of the head.

Command execution induced by eye movements and lid movements as well asmovements induced by the peri-orbital muscles, facial movements and headmovements can be advantageous in environments where an operator does nothave his hands available to type on a keyboard or to execute commands ona touchpad or other hand-computer interface. Such situations include,but are not limited, to industrial applications including automotive andairplane manufacturing, chip manufacturing, medical or surgicalprocedures and many other potential applications.

In some embodiments, the optical head mount display can include an eyetracking system. Different types of eye tracking systems can beutilized. The examples provided below are in no way thought to belimiting to the invention. Any eye tracking system known in the art nowcan be utilized.

Eye movement can be divided into fixations and saccades—when the eyegaze pauses in a certain position, and when it moves to anotherposition, respectively. The resulting series of fixations and saccadescan be defined as a scan path. The central one or two degrees of thevisual angle provide most of the visual information; the input from theperiphery is less informative. Thus, the locations of fixations along ascan path show what information locations were processed during an eyetracking session, for example during a surgical procedure.

Eye trackers can measure rotation or movement of the eye in severalways, for example via measurement of the movement of an object (forexample, a form of contact lens) attached to the eye, optical trackingwithout direct contact to the eye, and measurement of electricpotentials using electrodes placed around the eyes.

If an attachment to the eye is used, it can, for example, be a specialcontact lens with an embedded mirror or magnetic field sensor. Themovement of the attachment can be measured with the assumption that itdoes not slip significantly as the eye rotates. Measurements with tightfitting contact lenses can provide very accurate measurements of eyemovement. Additionally, magnetic search coils can be utilized whichallow measurement of eye movement in horizontal, vertical and torsiondirection.

Alternatively, non-contact, optical methods for measuring eye motion canbe used. With this technology, light, optionally infrared, can bereflected from the eye and can be sensed by an optical sensor or a videocamera. The information can then be measured to extract eye rotationand/or movement from changes in reflections. Optical sensor orvideo-based eye trackers can use the corneal reflection (the so-calledfirst Purkinje image) and the center of the pupil as features to track,optionally over time. A more sensitive type of eye tracker, thedual-Purkinje eye tracker, uses reflections from the front of the cornea(first Purkinje image) and the back of the lens (fourth Purkinje image)as features to track. An even more sensitive method of tracking is toimage features from inside the eye, such as the retinal blood vessels,and follow these features as the eye rotates and or moves. Opticalmethods, particularly those based on optical sensors or video recording,can be used for gaze tracking.

In some embodiments, optical or video-based eye trackers can be used. Acamera focuses on one or both eyes and tracks their movement as theviewer performs a function such as a surgical procedure. The eye-trackercan use the center of the pupil for tracking. Infrared or near-infrarednon-collimated light can be utilized to create corneal reflections. Thevector between the pupil center and the corneal reflections can be usedto compute the point of regard on a surface or the gaze direction.Optionally, a calibration procedure can be performed at the beginning ofthe eye tracking.

Bright-pupil and dark-pupil eye tracking can be employed. Theirdifference is based on the location of the illumination source withrespect to the optics. If the illumination is co-axial relative to theoptical path, then the eye acts is retroreflective as the light reflectsoff the retina creating a bright pupil effect similar to a red eye. Ifthe illumination source is offset from the optical path, then the pupilappears dark because the retroreflection from the retina is directedaway from the optical sensor or camera.

Bright-pupil tracking can have the benefit of greater iris/pupilcontrast, allowing more robust eye tracking with all iris pigmentation.It can also reduce interference caused by eyelashes. It can allow fortracking in lighting conditions that include darkness and very brightlighting situations.

The optical tracking method can include tracking movement of the eyeincluding the pupil as described above. The optical tracking method canalso include tracking of the movement of the eye lids and alsoperiorbital and facial muscles.

In some embodiments, the eye-tracking apparatus is integrated in anoptical head mounted display. In some embodiments, head motion can besimultaneously tracked, for example using a combination ofaccelerometers and gyroscopes forming an inertial measurement unit (seebelow).

In some embodiments, electric potentials can be measured with electrodesplaced around the eyes. The eyes generate an electric potential field,which can also be detected if the eyes are closed. The electricpotential field can be modelled to be generated by a dipole with thepositive pole at the cornea and the negative pole at the retina. It canbe measured by placing two electrodes on the skin around the eye. Theelectric potentials measured in this manner are called anelectro-oculogram.

If the eyes move from the center position towards the periphery, theretina approaches one electrode while the cornea approaches the opposingone. This change in the orientation of the dipole and consequently theelectric potential field results in a change in the measuredelectro-oculogram signal. By analyzing such changes eye movement can beassessed. Two separate movement directions, a horizontal and a vertical,can be identified. If a posterior skull electrode is used, a EOGcomponent in radial direction can be measured. This is typically theaverage of the EOG channels referenced to the posterior skull electrode.The radial EOG channel can measure saccadic spike potentials originatingfrom extra-ocular muscles at the onset of saccades.

EOG can be limited for measuring slow eye movement and detecting gazedirection. EOG is, however, well suited for measuring rapid or saccadiceye movement associated with gaze shifts and for detecting blinks.Unlike optical or video-based eye-trackers, EOG allows recording of eyemovements even with eyes closed. The major disadvantage of EOG is itsrelatively poor gaze direction accuracy compared to an optical or videotracker. Optionally, both methods, optical or video tracking and EOG,can be combined in select embodiments of the invention.

A sampling rate of 15, 20, 25, 30, 50, 60, 100, 120, 240, 250, 500, 1000Hz or greater can be used. Any sampling frequency is possibly. In manyembodiments, sampling rates greater than 30 Hz will be preferred.

Measuring Location, Orientation, Acceleration

The location, orientation, and acceleration of the human head, portionsof the human body, e.g. hands, arms, legs or feet, as well as portionsof the patient's body, e.g. the patient's head or extremities, includingthe hip, knee, ankle, foot, shoulder, elbow, hand or wrist and any otherbody part, can, for example, be measured with a combination ofgyroscopes and accelerometers. In select applications, magnetometers mayalso be used. Such measurement systems using any of these components canbe defined as inertial measurement units (IMU).

As used herein, the term IMU relates to an electronic device that canmeasure and transmit information on a body's specific force, angularrate, and, optionally, the magnetic field surrounding the body, using acombination of accelerometers and gyroscopes, and, optionally,magnetometers. An IMU or components thereof can be coupled with orregistered with a navigation system or a robot, for example byregistering a body or portions of a body within a shared coordinatesystem. Optionally, an IMU can be wireless, for example using WiFinetworks or Bluetooth networks.

Pairs of accelerometers extended over a region of space can be used todetect differences (gradients) in the proper accelerations of frames ofreferences associated with those points. Single- and multi-axis modelsof accelerometer are available to detect magnitude and direction of theacceleration, as a vector quantity, and can be used to sense orientation(because direction of weight changes), coordinate acceleration (so longas it produces g-force or a change in g-force), vibration, shock.Micromachined accelerometers can be utilized in some embodiments todetect the position of the device or the operator's head.

Piezoelectric, piezoresistive and capacitive devices can be used toconvert the mechanical motion into an electrical signal. Piezoelectricaccelerometers rely on piezoceramics or single crystals Piezoresistiveaccelerometers can also be utilized. Capacitive accelerometers typicallyuse a silicon micro-machined sensing element.

Accelerometers used in some of the embodiments can include small microelectro-mechanical systems (MEMS), consisting, for example, of littlemore than a cantilever beam with a proof mass.

Optionally, the accelerometer can be integrated in the optical headmounted devices and both the outputs from the eye tracking system andthe accelerometer(s) can be utilized for command execution.

With an IMU, the following exemplary information can be captured aboutthe operator and the patient and respective body parts: Speed, Velocity,Acceleration, Position in space, Positional change, Alignment,Orientation, and/or Direction of movement (e.g. through sequentialmeasurements)

Operator and/or patient body parts about which such information can betransmitted by the IMU include, but are not limited to: Head, Chest,Trunk, Shoulder, Elbow, Wrist, Hand, Fingers, Arm, Hip, Knee, Ankle,Foot, Toes, Leg, Inner organs, e.g. brain, heart, lungs, liver, spleen,bowel, bladder etc.

Any number of IMUs can be placed on the OHMD, the operator and/or thepatient and, optionally, these IMUs can be cross-referenced to eachother within a single or multiple coordinate systems or, optionally,they can be cross-referenced in relationship to an OHMD, a second andthird or more OHMDs, a navigation system or a robot and one or morecoordinate systems used by such navigation system and/or robot. Anavigation system can be used in conjunction with an OHMD without theuse of an IMU. For example, navigation markers including infraredmarkers, retroreflective markers, RF markers can be attached to an OHMDand, optionally, portions or segments of the patient or the patient'sanatomy. The OHMD and the patient or the patient's anatomy can becross-referenced in this manner or registered in one or more coordinatesystems used by the navigation system and movements of the OHMD or theoperator wearing the OHMD can be registered in relationship to thepatient within these one or more coordinate systems. Once the virtualdata and the live data of the patient and the OHMD are registered in thesame coordinate system, e.g. using IMUs, optical markers, navigationmarkers including infrared markers, retroreflective markers, RF markers,and any other registration method described in the specification orknown in the art, any change in position of any of the OHMD inrelationship to the patient measured in this fashion can be used to movevirtual data of the patient in relationship to live data of the patient,so that the visual image of the virtual data of the patient and the livedata of the patient seen through the OHMD are always aligned,irrespective of movement of the OHMD and/or the operator's head and/orthe operator wearing the OHMD. Similarly, when multiple OHMDs are used,e.g. one for the primary surgeon and additional ones, e.g. two, three,four or more, for other surgeons, assistants, residents, fellows, nursesand/or visitors, the OHMDs worn by the other staff, not the primarysurgeon, will also display the virtual representation(s) of the virtualdata of the patient aligned with the corresponding live data of thepatient seen through the OHMD, wherein the perspective of the virtualdata that is with the patient and/or the surgical site for the location,position, and/or orientation of the viewer's eyes for each of the OHMDsused and each viewer. The foregoing embodiments can be achieved sincethe IMUs, optical markers, RF markers, infrared markers and/ornavigation markers placed on the operator and/or the patient as well asany spatial anchors can be registered in the same coordinate system asthe primary OHMD and any additional OHMDs. The position, orientation,alignment, and change in position, orientation and alignment inrelationship to the patient and/or the surgical site of each additionalOHMD can be individually monitored thereby maintaining alignment and/orsuperimposition of corresponding structures in the live data of thepatient and the virtual data of the patient for each additional OHMDirrespective of their position, orientation, and/or alignment inrelationship to the patient and/or the surgical site.

Referring to FIG. 1, a system 10 for using multiple OHMDs 11, 12, 13, 14for multiple viewer's, e.g. a primary surgeon, second surgeon, surgicalassistant(s) and/or nurses(s) is shown. The multiple OHMDs can beregistered in a common coordinate system 15 using anatomic structures,anatomic landmarks, calibration phantoms, reference phantoms, opticalmarkers, navigation markers, and/or spatial anchors, for example likethe spatial anchors used by the Microsoft Hololens. Pre-operative data16 of the patient can also be registered in the common coordinate system15. Live data 18 of the patient, for example from the surgical site,e.g. a spine, optionally with minimally invasive access, a hiparthrotomy site, a knee arthrotomy site, a bone cut, an altered surfacecan be measured, for example using one or more IMUs, optical markers,navigation markers, image or video capture systems and/or spatialanchors. The live data 18 of the patient can be registered in the commoncoordinate system 15. Intra-operative imaging studies 20 can beregistered in the common coordinate system 15. OR references, e.g. an ORtable or room fixtures can be registered in the common coordinate system15 using, for example, optical markers IMUs, navigation markers orspatial mapping 22. The pre-operative data 16 or live data 18 includingintra-operative measurements or combinations thereof can be used todevelop, generate or modify a virtual surgical plan 24. The virtualsurgical plan 24 can be registered in the common coordinate system 15.The OHMDs 11, 12, 13, 14 can project digital holograms of the virtualdata or virtual data into the view of the left eye using the viewposition and orientation of the left eye 26 and can project digitalholograms of the virtual data or virtual data into the view of the righteye using the view position and orientation of the right eye 28 of eachuser, resulting in a shared digital holographic experience 30. Using avirtual or other interface, the surgeon wearing OHMD 1 11 can executecommands 32, e.g. to display the next predetermined bone cut, e.g. froma virtual surgical plan or an imaging study or intra-operativemeasurements, which can trigger the OHMDs 11, 12, 13, 14 to projectdigital holograms of the next surgical step 34 superimposed onto andaligned with the surgical site in a predetermined position and/ororientation.

Virtual data of the patient can be projected superimposed onto live dataof the patient for each individual viewer by each individual OHMD fortheir respective view angle or perspective by registering live data ofthe patient, e.g. the surgical field, and virtual data of the patient aswell as each OHMD in a common, shared coordinate system. Thus, virtualdata of the patient including aspects of a virtual surgical plan canremain superimposed and/or aligned with live data of the patientirrespective of the view angle or perspective of the viewer andalignment and/or superimposition can be maintained as the viewer moveshis or her head or body.

Novel User Interfaces

One subject of the present invention is to provide a novel userinterface where the human eye including eye movements and lid movementsincluding movements induced by the orbital and peri-orbital and selectskull muscles are detected by the eye tracking system and are processedto execute predefined, actionable computer commands.

An exemplary list of eye movements and lid movements that can bedetected by the system is provided in Table 1.

Table 1: Exemplary list of eye movements and lid movements detected bythe eye tracking software

-   -   1 blink    -   2 blinks    -   3 blinks    -   Fast blink, for example less than 0.5 seconds    -   Slow blink, for example more than 1.0 seconds    -   2 or more blinks with fast time interval, e.g. less than 1        second    -   2 or more blinks with long time interval, e.g. more than 2        seconds (typically chosen to be less than the natural time        interval between eye blinks)    -   Blink left eye only    -   Blink right eye only    -   Blink left eye and right eye simultaneously    -   Blink left eye first, then within short time interval (e.g. less        than 1 second), blink right eye    -   Blink right eye first, then within short time interval (e.g.        less than 1 second), blink left eye    -   Blink left eye first, then within long time interval (e.g. more        than 2 seconds), blink right eye    -   Blink right eye first, then within long time interval (e.g. more        than 2 seconds), blink left eye    -   Rapid eye movement to left    -   Rapid eye movement to right    -   Rapid eye movement up    -   Rapid eye movement down    -   Widen eyes, hold for short time interval, e.g. less than 1        second    -   Widen eyes, hold for long time interval, e.g. more than 2        seconds    -   Close both eyes for 1 second etc.    -   Close both eyes for 2 seconds or more etc.    -   Close both eyes, hold, then open and follow by fast blink    -   Close left eye only 1 second, 2 seconds etc.    -   Close right eye only 1 second, 2 seconds etc.    -   Close left eye, then right eye    -   Close right eye, then left eye    -   Blink left eye, then right eye    -   Blink right eye, then left eye    -   Stare at field, virtual button for 1, 2, 3 or more seconds;        activate function, e.g. Zoom in or Zoom out

Any combination of blinks, eye movements, sequences, and time intervalsis possible for encoding various types of commands. These commands canbe computer commands that can direct or steer, for example, a surgicalinstrument or a robot.

The invention provides also for methods of executing commands by way offacial movements and movements of the head.

An exemplary list of facial movements and head movements that can bedetected by the system is provided in Table 2. (This list is only anexample and by no way meant to be exhaustive; any number or combinationof movements is possible).

Table 2: Exemplary list of facial movements and head movements detected:

-   -   Move head fast to right and hold    -   Move head fast to left and hold    -   Move head fast down and hold    -   Move head fast down and hold    -   Move head fast to right and back    -   Move head fast to left and back    -   Move head fast down and back    -   Move head fast down and back    -   Tilt head to left and hold    -   Tilt head to right and hold    -   Tilt head to left and back    -   Tilt head to right and back    -   Open mouth and hold    -   Open mouth and close    -   Twitch nose once    -   Twitch nose twice etc.

Exemplary commands executed using eye movements, lid movements, facialmovements and head movements are listed in Table 3.

Table 3: Exemplary list of commands that can be executed by tracking eyemovement, lid movement, facial movement and head movement (this list isonly an example and by no way meant to be exhaustive; any number orcombination of commands is possible; application specific commands canbe executed in this manner as well).

-   -   Click    -   Point    -   Move pointer        -   Slow        -   Fast    -   Scroll, e.g. through images        -   Fast scroll        -   Slow scroll    -   Scroll up    -   Scroll down    -   Scroll left    -   Scroll right    -   Drag    -   Swoosh    -   Register    -   Toggle 2D vs. 3D    -   Switch imaging study    -   Overlay images    -   Fuse images    -   Register images    -   Cut    -   Paste    -   Copy    -   Undo    -   Redo    -   Delete    -   Purchase    -   Provide credit card information    -   Authorize    -   Go to shopping card    -   OHMD on    -   OHMD off    -   Eye tracking on    -   Eye tracking off    -   Eye command execution on    -   Eye command execution off    -   Facial command execution on    -   Facial command execution off    -   Turn surgical instrument on (e.g. oscillating saw, laser etc.)    -   Turn surgical instrument off    -   Increase intensity, speed, energy deposed of surgical instrument    -   Reduce intensity, speed, energy deposed of surgical instrument    -   Change direction of surgical instrument    -   Change orientation of surgical instrument    -   Change any type of setting surgical instrument

In some embodiments of the invention, eye movements, lid movements,facial movement, head movements alone or in combination can be used tosignal numerical codes or sequences of numbers or sequences of machineoperations. Such sequences of numbers can, for example, be used toexecute certain machine operating sequences.

Head Movement to Control Movement of a Surgical Instrument

In some embodiments of the invention, head movement can be used tocontrol a surgical instrument. For example, in a robot assistedprocedure with haptic feedback from the robot, the surgeon can use hisor her hands in controlling the direction of a surgical instrument. Thesurgeon can move the head forward. This forward motion is captured by anIMU and translated into a forward movement of a robotic arm holding asurgical instrument along the direction of the surgical instrument. Abackward movement of the head can be captured by the IMU and can betranslated into a backward movement of the robotic arm holding asurgical instrument along the direction of the surgical instrument.

In some embodiments of the invention, eye movements, lid movements,facial movement, head movements alone or in combination can be used tosignal Morse codes. The International Morse Code encodes the Latinalphabet using a small set of punctuation and procedural signals asstandardized sequences of short and long signals called dots and dashes.Each character (letter or numeral) is represented by a unique sequenceof dots and dashes. The duration of a dash is three times the durationof a dot. Each dot or dash is followed by a short silence, equal to thedot duration. The letters of a word are separated by a space equal tothree dots (one dash), and the words are separated by a space equal toseven dots.

An example how Morse code can be executed using eye commands is providedas follows; this is in no way meant to be limiting. Many differentimplementations are possible. A dot can be executed, for example, usinga fast blink of both eyes (typically less than 1 sec), while a dash canbe executed by closing the right eye only, for example for one second.The letter A in Morse code is a dot followed by a dash. With thisencoding of Morse code, the letter A can be executed with a fast blinkof both eyes (dot), followed by closing the right eye only for onesecond (dash). The letter B (dash, three dots), can be executed byclosing the right eye only for one second (dash) followed by three fastblinks of both eyes (three dots) and so forth. Letters can be separated,for example, by maintaining a two second or longer break between eyecommands. Alternatively, in another example, letters can be separate byclosing only the left eye for about one second.

Binary codes can optionally also be executed using eye commands. Forexample, a fast blink of both eyes can represent the number 0, whileclosing the right eye only for about one second can represent thenumber 1. Alternatively, closing the right eye only for about one secondcan represent the number 0, while closing the left eye only for aboutone second can represent the number 1. Many different types of encodingare possible. Other numericals can also be executed using, for example,some of the eye, lid, facial and/or head movements shown in Tables 1 and2.

Many different languages can be executed in this fashion. These include,optionally, also computer languages, e.g. Fortran, Pascal, C, C++, C−−,Basic and many others known in the art. In some embodiments of theinvention, eye, lid, facial and head movement commands can be paired orused in conjunction with voice commands, hand commands, gesturecommands, keyboard commands, track pad commands, mouse commands,graphical user interface commands and any other command input deviceknown in the art. The OHMD can optionally also include one or more touchsensitive sensors.

In select environments, eye commands add benefit of being able tonavigate a screen or execute commands while maintaining privacy orconfidentiality related to the commands. For example, in a hospitalenvironment, with other patients or visitors nearby, eye commands can beutilized to access a patient's medical records or to order lab tests orother diagnostic tests without bystanders being aware that these recordsare being reviewed or that these tests are being ordered.

At a conference, the wearer of an optical head mounted display canutilize eye commands to turn on a video or audio recording function ortransmission to a remote site or remote conference room withoutdisclosing that the recording function has been activated. This is quitedifferent from manual activation of a recording function, where the userwould, for example, push a button or a touch sensitive sensor on theoptical head mounted display in order to activate the recordingfunction.

In some embodiments, a user can utilize eye movements, facial movementsor head movements to direct digital camera for taking photographs orvideos. Commands can include but are not limited to zoom in, zoom out,move region of interest left, right up, down, take photo, take sequenceof photos, turn on/off flash start video recording, stop videorecording, change resolution, increase resolution, decrease resolution.

Any other camera command known in the art can be executed in this mannerusing eye movement, facial movement or head movement based commands. Byutilizing one or more commands of this type, the user can maintainprivacy while obtaining image information about the surroundingenvironment.

Eye commands can be useful to surgeons or operating room personnel toexecute commands without use of the hands and thereby maintainingsterility.

Fusing Physical World with Imaging and Other Data of a Patient

In some embodiments of the invention, an operator such as a surgeon maylook through an OHMD observing physical data or information on apatient, e.g. a surgical site or changes induced on a surgical site,while pre-existing data of the patient are superimposed onto thephysical visual representation of the live patient.

The pre-existing data of the patient can be an imaging test or imagingdata or other types of data including metabolic information orfunctional information.

The pre-existing data of the patient including one or more imaging testsor other types of data including metabolic or functional information canbe obtained at a time different from the time of the surgical procedure.For example, the pre-existing data of the patient can be obtained one,two, three or more days or weeks prior to the surgical procedure.

The pre-existing data of the patient including one or more imaging testsor other types of data including metabolic or functional information aretypically obtained with the patient or the surgical site being locatedin a different location or a different object coordinate system in thepre-existing data when compared to the location or the object coordinatesystem of the live patient or the surgical site in the live patient.Thus, pre-existing data of the patient or the surgical site aretypically located in a first object coordinate system and live data ofthe patient or the surgical site are typically located in a secondobject coordinate systems; the first and the second object coordinatesystem are typically different from each other. The first objectcoordinate system with the pre-existing data needs to be registered withthe second object coordinate system with the live data of the patientincluding, for example, the live surgical site.

Scan Technology

The following is an exemplary list of scanning and imaging techniquesthat can be used or applied for various aspects of the invention; thislist is not exhaustive, but only exemplary. Anyone skilled in the artcan identify other scanning or imaging techniques that can be used inpracticing the invention.

For a detailed description of these different scanning and imagingtechniques, see for example, Bushberg et al. The Essential Physics ofMedical Imaging, 3^(rd) edition, Wolters, Kluwer, Lippincott, 2012.

-   -   X-ray imaging, 2D, 3D, supine, upright or in other body        positions and poses, including analog and digital x-ray imaging    -   Digital tomosynthesis    -   Cone beam CT    -   Ultrasound    -   Doppler ultrasound    -   Elastography, e.g. using ultrasound or MRI    -   CT    -   MRI        -   including, for example, fMRI, diffusion imaging, stroke            imaging, MRI with contrast media    -   Functional MRI (fMRI), e.g. for brain imaging and functional        brain mapping    -   Magnetic resonance spectroscopy    -   PET    -   SPECT-CT    -   PET-CT    -   PET-MRI    -   Upright scanning, optionally in multiple planes or in 3D using        any of the foregoing modalities, including x-ray imaging,        ultrasound etc.    -   Contrast media        -   e.g. iodinated contrast agents for x-ray and CT scanning, or            MRI contrast agents.        -   contrast agents can include antigens or antibodies for cell            or tissue specific targeting        -   other targeting techniques, e.g. using liposomes, can also            be applied        -   molecular imaging            -   To highlight metabolic abnormalities in the brain and                target surgical instruments towards area of metabolic                abnormality        -   any contrast agent known in the art can be used in            conjunction with the invention.

Multi-Dimensional Imaging, Reconstruction and Visualization

Various embodiments of this invention can be practiced in one, two,three or more dimensions. The following is an exemplary list ofpotential dimensions, views, projections, angles, or reconstructionsthat can be applied; this list is not exhaustive, but only exemplary.Anyone skilled in the art can identify additional dimensions, views,projections, angles or reconstructions that can be used in practicingthe invention. Exemplary dimensions are listed in Table 4.

TABLE 4: Exemplary list of potential dimensions, views, projections,angles, or reconstructions that can be displayed using virtualrepresentations with optical head mounted display(s), optionallystereoscopic

1^(st) dimension: superoinferior, e.g. patient physical data

2^(nd) dimension: mediolateral, e.g. patient physical data

3^(rd) dimension: anteroposterior, e.g. patient physical data

4^(th)-6^(th) dimension: head motion (and with it motion ofglasses/OHMD) in 1, 2 or 3 dimensions

7^(th)-9^(th) dimension: instrument motion in 1, 2 or 3 dimensions, e.g.in relationship to surgical field, organ or head including head motion

10^(th)-13^(th) dimension: arm or hand motion in 1, 2 or 3 dimensions,e.g. in relationship to surgical field, organ or head including headmotion

14^(th)-16^(th) dimension: virtual 3D data of patient, obtained, forexample from a scan or intraoperative measurements

17^(th)-19^(th) dimension: vascular flow; in 1, 2 or 3 dimensions, e.g.in relationship to surgical field, organ or head including head motion

20^(th)-22^(nd) dimension: temperature map (including changes induced bycryo- or hyperthermia), thermal imaging, in 1, 2 or 3 dimensions, e.g.in relationship to surgical field

25^(th)-28^(th) dimension: metabolic map (e.g. using MRS, PET-CT,SPECT-CT), in 1, 2 or 3 dimensions, e.g. in relationship to surgicalfield

29^(th)-32^(nd) dimension: functional map (e.g. using fMRI, PET-CT,SPECT-CT, PET, kinematic imaging), in 1, 2 or 3 dimensions, e.g. inrelationship to surgical field or patient

Any oblique planes are possible. Any perspective projections arepossible. Any oblique angles are possible. Any curved planes arepossible. Any curved perspective projections are possible. Anycombination of 1D, 2D, and 3D data between the different types of datais possible.

Registering Virtual Data with Live Data Seen Through Optical HeadMounted Display

In some embodiments, virtual data of a patient can be superimposed ontolive data seen through the optical head mounted display. The virtualdata can be raw data in unprocessed form, e.g. preoperative images of apatient, or they can be processed data, e.g. filtered data or segmenteddata.

Data Segmentation

When images of the patient are superimposed onto live data seen throughthe optical head mounted display, in many embodiments image segmentationcan be desirable. Any known algorithm in the art can be used for thispurpose, for example thresholding, seed point techniques, live wire,deformable models, statistical models, active shape models, level setmethods, marching cubes algorithms, artificial neural networks, deeplearning techniques, or combinations thereof and the like. Many of thesealgorithms are available is part of open-source or commercial libraries,for instance the Insight Segmentation and Registration Toolkit (ITK),the Open Source Computer Vision Library OpenCV, G'MIC (GREYC's Magic forImage Computing), Caffe, or MATLAB (MathWorks, Natick, Mass.). Arepresentative workflow for segmentation and subsequent is provided inFIG. 2. An optional pre-operative imaging study 40 can be obtained. Anoptional intra-operative imaging study 41 can be obtained. Thepre-operative 40 or intra-operative 41 imaging study can be segmented42, extracting, for example, surfaces, volumes or key features. Anoptional 3D reconstruction or 3D rendering 43 can be generated. Thepre-operative 40 or intra-operative 41 imaging study and any 3Dreconstruction or 3D rendering 43 can be registered in a commoncoordinate system 44. The pre-operative 40 or intra-operative 41 imagingstudy and any 3D reconstruction or 3D rendering 43 can be used forgenerating a virtual surgical plan 45. The virtual surgical plan 45 canbe registered in the common coordinate system 44. The surgical site 46can be registered in the common coordinate system 44. Intra-operativemeasurements 47 can be obtained and can be used for generating a virtualsurgical plan 45. An optical head mounted display 48 can project ordisplay digital holograms of virtual data or virtual data 49superimposed onto and aligned with the surgical site. The OHMD 48 isconfigured to use a built in camera or image capture or video capturesystem 50 to optionally detect and/or measure the position and/ororientation and/or alignment of one or more optical markers 51, whichcan be used for the coordinate measurements 52, which can be part of theintra-operative measurements 47.

Software and Algorithms for Registration

Registration of virtual data with live data can be performed using avariety of techniques know in the art. These include, but are notlimited to, surface registration algorithms such as the IterativeClosest Point algorithm, statistical models, Active Shape Models, mutualinformation-based or other volume registration algorithms, objectrecognition, pattern recognition or computer vision techniques, deeplearning or other artificial intelligence methods. The processed datacan, for example, consist of mesh data, parametric surface data, pointcloud data, volume data or a combination thereof. These methods areknown in the art and have been implemented in publicly and/orcommercially available code libraries and application programminginterfaces (API's), such as the Insight Segmentation and RegistrationToolkit (ITK), the open-source computer vision library OpenCV, Elastix,Plastimatch, or the Medical Image Registration Toolkit (MIRTK).

Superimposition of Virtual Data and Live Data by the OHMD

In some embodiments, segmented data or raw data can be superimposed onthe patient's live data seen through the optical head mounted display.This superimposition can occur in unregistered form, i.e. the patient'svirtual data may not be aligned with the live data seen through theoptical head mounted display. In this case, the operator who is wearingthe OHMD may move his/her head in a direction of orientation that willsuperimpose corresponding features of virtual data and live patientdata. The surgeon or operator can also move and re-orient the virtualdata using other means, e.g. a trackball or a virtual display interfacedisplayed in the OHMD, unrelated to the surgeon/operator head movement.The operator can adjust the magnification of the live data so that thesize, shape, length, thickness of certain features of the virtual datamatches that of the live data for a given distance to theobject/patient.

For example, during brain surgery, the surgeon may visually in live datalook at the exposed gyri and sulci of the patient's brain. The OHMD candisplay a virtual 3D model of the gyri and sulci of the patient. Thesurgeon can optionally adjust the magnification of the 3D model so thatthe model will match the size or width or the length of thecorresponding gyri and sulci in the live data. The surgeon canoptionally adjust the transparency or opacity of the virtual datadisplayed in the OHMD. The ratio of virtual vs. live data transmittedthrough the OHMD can be 1:10, 1:9, 1:8, 1:5, 1:2, 1:1, 2:1, 3:1, 5:1,8:1, 10:1, as well as fractions or multiples thereof. Any combination oftransparency or opacity of virtual data and live data is possible. Thesurgeon can move his/her head in a direction or orientation that willsuperimpose virtual features, e.g. the patient's gyri and sulci, withthe live patient data.

Once the data have been superimposed, the surgeon can optionallyregister the virtual data with the live data. This registration can beas simple as described here, e.g. a visual confirmation from the surgeonthat virtual and live data are substantially matching or substantiallysuperimposed. At this time, the surgeon can optionally reference thevirtual data and/or the coordinate system of the virtual data in 2, 3 ormore dimensions with the live data and/or the coordinate system of thelive data. Once the data are registered, the surgeon can move his/herhead into any desired position or orientation, for example for viewingthe patient's brain or a lesion and adjacent, e.g. sensitive, anatomyfrom different view angles. The IMU of the OHMD will register the headmovement, the direction of the head movement, the new head position andhead orientation. The change in location and orientation of thesurgeon's head can be simultaneously or, if desired, non-simultaneouslyapplied to the virtual data which can now be superimposed with theresultant new position and orientation in relationship to the live data.In addition, when the surgeon moves his/her head or body further awayfrom the target anatomy, the change in position and the increase indistance from the target anatomy can be measured by the IMU. Dependingon the distance from the IMU, a magnification or minification factor canbe applied to the virtual data so that the size, shape and dimensions ofthe virtual data will, in some embodiments, be close to or match thesize, shape and dimensions of the live data, irrespective of thedistance, location and orientation of the surgeon's head.

For purposes of registration of virtual data and live data, the OHMD canbe optionally placed in a fixed position, e.g. mounted on a stand or ona tripod. While the OHMD is placed in the fixed position, live data canbe viewed by the surgeon and they can be, optionally recorded with acamera and/or displayed on a monitor. Virtual data can then besuperimposed and the matching and registration of virtual data and livedata can be performed. At this point, the surgeon or an operator canremove OHMD from the fixed position and the surgeon can wear the OHMDduring the surgical procedure.

The virtual data can optionally be displayed using a different color,e.g. red, green, yellow etc. Optionally, only the outline of selectfeatures of the virtual data may be displayed. For example, thesefeatures can be the sulci of the patient's brain (e.g. with a black lineor black or lines with other colors), with no visualization of the gyrithat these sulci border. Or, for example, only a lesion, e.g. a tumorsuch as, in the example of the brain, glioblastoma, can be displayed. Orcombinations of virtual data of normal tissue and pathologic tissue canbe displayed.

The virtual data can be registered with the live data seen through theoptical head mounted display. The registration can occur using anymethod known in the art for registering or cross-referencing virtual andlive data, in 2, 3, or more dimensions.

In some embodiments, the registration of the virtual data and the livedata will be maintained through the surgical procedure. In someembodiments, the registration of the virtual data and the live data willbe maintained during select portions of the surgical procedure or thesurgical plan, which can be or can include a virtual, e.g. apreoperatively generated, surgical plan.

In some embodiments of the invention, the superimposition of the virtualdata and the live data by the OHMD occurs simultaneously. In someembodiments, the superimposition of the virtual data and the live databy the OHMD is not simultaneous. For example, the virtual data can besuperimposed intermittently.

Virtual data can be transparent, translucent or opaque. If virtual dataare opaque, they may be displayed intermittently so that the operator orsurgeon can see how they project in relationship to the live data of thepatient.

If combinations of virtual data are displayed simultaneously with thelive data, the different types of virtual data can be displayed withdifferent colors. Representative combinations of virtual and live dataare provided below. The following is only illustrative in nature and byno means meant to be limiting of the invention:

Live data: the patient's brain; surgically exposed gyri and sulci.

Live data: surgical instrument, e.g. biopsy needle or cutting tool

Virtual data: the patient's brain with gyri and sulci derived andoptionally segmented from an imaging modality, e.g. a CT scan or an MRIscan

Virtual data: a brain tumor, deep seated inside the brain

Virtual data: the same surgical instrument currently used by thesurgeon, in a virtual representation of the instrument, the virtual dataindicating the desired orientation, location or direction of thesurgical instrument.

Any of the foregoing virtual data can be displayed in two dimensions orthree dimensions. Multi-dimensional displays as outlined in othersections of the specification are possible.

For example, the patient's normal tissue, e.g. normal brain tissue, canoptionally be displayed in two dimensions, e.g. using grey level images,while the patient's abnormal tissue, e.g. a stroke, a hemorrhage or atumor, can be displayed in three dimensions. Any combination of 2D, 3D,and multi-dimensional images is possible for display by the OHMD; anycombination of 2D, 3D, and multi-dimensional images can be superimposedon live patient data by the OHMD.

The virtual 2D, 3D, and multi-dimensional data can be generated oracquired by different data acquisition technologies, e.g. differentimaging tests etc.

Locking or Moving of Virtual Data

In some embodiments of the invention, virtual data can be locked inrelationship to the surgeon or operator or in relationship to thepatient or a certain target anatomy within a patient. This means even ifthe surgeon moves his or her head or the body or parts of the patient'sanatomy are being moved, the virtual data will not move in the OHMDdisplay. For example, once registration has occurred, the OHMD candisplay a virtual image of a target tissue or adjacent tissue. Thevirtual image of the target tissue or adjacent tissue can be, forexample, an image through a tumor or other type of pathologic tissue. Asthe surgeon or operator moves his or her head or body during thesurgical procedure, the virtual data will not move, but are beingdisplayed within the same location.

In some embodiments of the invention, virtual data can move inrelationship to the surgeon or operator or in relationship to thepatient or a certain target anatomy within a patient. This means if thesurgeon moves his or her head or the body or parts of the patient'sanatomy are being moved, the virtual data will move in the OHMD display.For example, once registration has occurred, the OHMD can display avirtual image of a target tissue or adjacent tissue. The virtual imageof the target tissue or adjacent tissue can be, for example, an imagethrough a tumor or other type of pathologic tissue. As the surgeon oroperator moves his or her head or body during the surgical procedure,the virtual data will move and change location and orientation the sameway how the surgeon moves his/her head or body, typically reflecting thechange in perspective or view angle that the surgeon obtained by movinghis or her head or body.

Optionally the moving of the virtual data can be at greater virtualdistance or greater angle or lesser virtual distance or lesser anglethan the movement of the surgeon's head or body.

Improving the Accuracy of Moving or Re-Orienting Virtual Data

Once registration between virtual data and physical data has occurred,the moving or re-orienting of virtual data to follow, for example, thesurgeon's head movements or body movements or operating arm or handmovements, or the movements of the patient or certain body parts of thepatient can be accomplished, for example, by monitoring the movement andchange in location and/or orientation of the surgeon's head using theIMU of the OHMD.

In some embodiments, optical or RF tracker's or other tracking devicesknown in the art can be applied to the OHMD and/or the patient includingselect body parts or target tissues of the patient, e.g. the patient'sknee. Using standard surgical navigation techniques known in the art,the spatial location of the optical or RF trackers can be recorded, forexample for a starting pose or position or location. Movement of thetrackers, e.g. induced by movement of the surgeon's head or body or bymovement of at least a part of the patient, can then be tracked usingthe navigation system. The information on positional change,orientational change or movement direction of the surgeon's head or thepatient or both can then be used to update the virtual data, or thedisplay of the virtual data in the OHMD, or both correspondingly. Inthis manner, the virtual data and the live data can be superimposed bythe OHMD, typically in an accurate manner.

Optionally, positional, orientational, directional data and the likegenerated by the IMU can be used in conjunction with such data generatedby a surgical navigation system. A combination of data can be beneficialfor more accurate measurement of changes in position or orientation ofthe surgeon's head, body, operating arm, hand, or the patient.

Use of Virtual Data in 2 or More Dimensions

In some embodiments of the invention, the OHMD can display a 2D virtualimage of the patient. The image can be a transmission type image, e.g.an x-ray or CT scout scan. The image can be a cross-sectional image ofselect anatomy of the patient. The image can be an original image or areformatted, reconstructed or segmented or partially segmented image ofthe patient.

In some embodiments of the invention, a surgeon will look through theOHMD at the patient's live data, e.g. the exposed brain surface with thepatient's gyri and sulci. The surgeon can register virtual data of thepatient, e.g. an MRI scan of the patient's brain, relative to thepatient's live data. Registration can occur in 2, 3 or more dimensions.Registration of virtual data in relationship to live data can includeregistration of different types of virtual data, e.g. different types ofnormal or diseased tissue, different imaging modalities used, differentdimensions used for different types of normal or diseased tissue etc.More than one 2D scan plane can be displayed simultaneously. These 2Dscan planes can be parallel or non-parallel, orthogonal ornon-orthogonal at variable angles.

Scrolling Through, Moving of Virtual Data Superimposed onto Live Data

In some embodiments of the invention, a surgeon or operator mayoptionally scroll through a set of consecutive or non-consecutivevirtual 2D image data as well as 3D image data which are beingsuperimposed onto the patient's live data, typically live data from thesame anatomic region, e.g. a brain, a spine, a hip, a knee etc. Thescrolling can be directed through any type of user interface, known inthe art. For example, a surgeon can use a virtual interface projected bythe OHMD where he or she can move a virtual arrow up or down or left orright to scroll the images backward or forward or, for example, torotate the images or to display them in different multiplanar angles orto change the view angle or projection angle.

Optionally, the surgeon can scroll through the virtual image data ormove virtual image data by moving his head back and forth, e.g. forscrolling backward or forward in a virtual image volume. The surgeon canmove his or her head left or right for example, to rotate the images orto display them in different multiplanar angles or to change the viewangle or projection angle of a 3D image.

Optionally, the surgeon can scroll through the virtual image data bymoving his or her hand or finger or any other body part back and forth,e.g. for scrolling backward or forward in a virtual image volume. Thesurgeon can move his or her hand or finger or any other body part backand forth left or right for example, to rotate the images or to displaythem in different multiplanar angles or to change the view angle orprojection angle. The surgeon can move his or her hand or finger in aspinning or rotating movement to spin or rotate the virtual data. Anycombination of head or hand or eye and other body signals can be usedfor changing the display of the virtual data.

Optionally, these display changes of the virtual data can be executed inthe OHMD using the same location, position, orientation, angular,direction and movement related changes that are made by the surgeon'sbody part used to trigger the change in display. Alternatively, any oneof location, position, orientation, angular, direction and movementrelated changes of the virtual data can be executed using amagnification factor or a minification factor in relationship to thechanges in location, position, orientation, angular, direction andmovement of the surgeon's body part. These magnification or minificationfactors can be linear or non-linear, e.g. exponential or logarithmic. Insome embodiments, the further the surgeon's body part controlling themovement of the virtual data in the OHMD display moves away from itsoriginal position, the greater the induced change on the movement of thevirtual data in the OHMD. In some embodiments, the further the surgeon'sbody part controlling the movement of the virtual data in the OHMDdisplay moves away from its original position, the smaller the inducedchange on the movement of the virtual data in the OHMD.

Use of Virtual Data in 3 or More Dimensions

In some embodiments of the invention, the OHMD can display a 3D virtualimage of the patient. A 3D representation of the patient can include a3D display of different types of anatomy, for example in an area ofintended surgery or a surgical site.

A 3D reconstruction of image data or other data of the patient can begenerated preoperatively, intraoperatively and/or postoperatively. Avirtual 3D representation can include an entire anatomic area or selecttissues or select tissues of an anatomic area. Different tissues can bevirtually displayed by the OHMD in 3D using, for example, differentcolors. Normal tissue(s) and pathologic tissue(s) can be displayed inthis manner.

Normal tissue can, for example, include brain tissue, heart tissue, lungtissue, liver tissue, vascular structures, bone, cartilage, spinaltissue, intervertebral disks, nerve roots. Any tissue can be visualizedvirtually by the OHMD.

Registration of Virtual Data and Live Data of a Patient, for ExampleOver a Surgical Site

In some embodiments of the invention virtual data of a patient displayedby an OHMD and live data of a patient seen through an OHMD are spatiallyregistered in relationship to each other, for example in a commoncoordinate system, for example with one or more optical OHMDs in thesame common coordinate system. Virtual and physical surgical instrumentsand implant components can also be registered in the common coordinatesystem. Spatial co-registration can have the benefit that thesimultaneous display of virtual and live data of the patient is notaffected or less affected when the surgeon moves his or her head orbody, when the OHMD moves or when the patient moves. Thus, the viewperspective of the live data of the patient seen by the surgeon's eyesthrough the OHMD, e.g. the live surgical field, can stay the same as theview perspective of the virtual data of the patient seen by thesurgeon's eyes through the display of the OHMD unit, e.g. the virtualsurgical field, virtual surgical plane, virtual paths, virtual cut pathsor planes, projected into the surgeon's eyes, even as the surgeon moveshis or her head or body. In this manner, the surgeon does not need tore-think or adjust his hand eye coordination since live data of thepatient seen through the surgeon's eye and virtual data of the patientseen through the OHMD display are superimposed, which is fundamentallydifferent from other approaches such as surgical navigation which employa separate computer monitor in the OR with a view angle for the surgeonthat is different than his or her view angle for the live data of thepatient and the surgical field. Also, with surgical navigation, a firstvirtual instrument can be displayed on a computer monitor which is arepresentation of a physical instrument tracked with navigation markers,e.g. infrared or RF markers, and the position and/or orientation of thefirst virtual instrument can be compared with the position and/ororientation of a corresponding second virtual instrument generated in avirtual surgical plan. Thus, with surgical navigation the positionsand/or orientations the first and the second virtual instruments arecompared.

With guidance in mixed reality environment, e.g. with stereoscopicdisplay like an electronic holographic environment, a virtual surgicalguide, tool, instrument or implant can be superimposed onto the joint,spine or surgical site. Further, the physical guide, tool, instrument orimplant can be aligned with the 2D or 3D representation of the virtualsurgical guide, tool, instrument or implant. Thus, guidance in mixedreality environment does not need to use a plurality of virtualrepresentations of the guide, tool, instrument or implant and does notneed to compare the positions and/or orientations of the plurality ofvirtual representations of the virtual guide, tool, instrument orimplant.

In some embodiments, virtual data can move in relationship to thesurgeon or operator or in relationship to the patient or a certaintarget anatomy within a patient. This means if the surgeon moves his orher head or the body or parts of the patient's anatomy are being moved,the virtual data will move in the OHMD display. For example, onceregistration of the OHMD, the virtual data of the patient and the livedata of the patient in a common coordinate system has occurred, the OHMDcan display a virtual image of a target tissue or adjacent tissue. Thevirtual image of the target tissue or adjacent tissue can be, forexample, an image of or through a tumor or other type of pathologictissue or a spine or a spinal pedicle. As the surgeon or operator moveshis or her head or body during the surgical procedure, the virtual datawill move and change location and orientation the same way how thesurgeon moves his/her head or body, typically reflecting the change inperspective or view angle that the surgeon obtained by moving his or herhead or body. The virtual data can include a 3D representation of asurgical tool or instrument such as a needle for kyphoplasty orvertebroplasty, where the virtual representation of the needle shows itsintended location, orientation or path in relationship to the spineand/or a pedicle. The virtual data can also include a medical device,such as a pedicle screw, wherein the virtual data of the pedicle screwshows its intended location, orientation or path in relationship to thespine, and/or a pedicle, and/or a vertebral body.

In some embodiments, registration is performed with at least three ormore points that can be superimposed or fused into a common objectcoordinate system for virtual data and live data. Registration can alsobe performed using a surface or a 3D shape of an anatomic structurepresent in both virtual data and live data of the patient. In this casethe virtual surface can be moved until it substantially matches the livesurface of the patient or the virtual shape can be moved until itsubstantially matches the live shape of the patient.

Registration of virtual data of a patient and live data of a patient canbe achieved using different means. The following is by no means meant toby limiting of the invention, but is only exemplary in nature.

Registration of Virtual Patient Data and Live Patient Data UsingDirectly or Indirectly Connected Object Coordinate Systems

Registration of virtual and live data of the patient can be performed ifthe virtual data, e.g. imaging data of the patient, are acquired withthe patient located in a first object coordinate system and the livedata, e.g. during surgery, are observed or acquired with the patientlocated in a second object coordinate system, wherein the first and thesecond object coordinate system can be connected by direct, e.g.physical, or indirect, e.g. non-physical, means. A direct connection ofthe first and second object coordinate system can be, for example, aphysical connection between the first and second object coordinatesystem. For example, the patient can be moved from the first to thesecond object coordinate system along the length of a tape measure. Orthe patient can be scanned inside a scanner, e.g. a CT scanner or MRIscanner, and the scanner table can be subsequently moved out of thescanner for performing a surgical procedure with the patient stilllocated on the scanner table. In this case, the scanner table can be aform of physical connection between the first and the second objectcoordinate system and the length of the table movement between the scanposition and the outside the scanner position (for the live data, e.g.the surgical procedure) can define the coordinate transformation fromthe first to the second object coordinate system.

An indirect connection between the first (virtual data) and second (livedata) object can be established if the patient is moved between theacquiring the virtual data, e.g. using an imaging test, and the livedata, e.g. while performing a surgical procedure, along a defined path,wherein the direction(s) and angle(s) of the path are known so that thefirst and the second object coordinate system can be cross-referencedand an object coordinate transfer can be applied using the knowninformation of the defined path and virtual data of the patient, livedata of the patient and the OHMD can be registered in a commoncoordinate system. Virtual and physical surgical instruments and implantcomponents can also be registered in the common coordinate system.

Registration of virtual patient data and live patient data is alsopossible without directly or indirectly connected object coordinatesystems using other means and methods as will be explained in thefollowing paragraphs and columns, for example when the patient performedone or more movements of unknown direction, length or magnitude.Combinations of all different registration methods described in thespecification are possible, e.g. for switching registration methodsduring a procedure or for simultaneously using multiple registrationmethods, e.g. for enhancing the accuracy of the registration.

Registration Using Spatial Mapping

Live data, e.g. live data of the patient, the position and/ororientation of a physical instrument, the position and/or orientation ofan implant component, the position and/or orientation of one or moreOHMDs, can be acquired or registered, for example, using a spatialmapping process. This process creates a three-dimensional meshdescribing the surfaces of one or more objects or environmentalstructures using, for example and without limitation, a depth sensor,laser scanner, structured light sensor, time of flight sensor, infraredsensor, or tracked probe. These devices can generate 3D surface data bycollecting, for example, 3D coordinate information or information on thedistance from the sensor of one or more surface points on the one ormore objects or environmental structures. The 3D surface points can thenbe connected to 3D surface meshes, resulting in a three-dimensionalsurface representation of the live data. The surface mesh can then bemerged with the virtual data using any of the registration techniquesdescribed in the specification.

The live data can be static, or preferably, it can be continuouslyupdated with additional information to incorporate changes in theposition or surface of the one or more objects or environmentalstructures. The additional information can, for example be acquired by adepth sensor, laser scanner, structured light sensor, time of flightsensor, infrared sensor, or tracked probe.

For initial spatial mapping and updating of mapping data, commonlyavailable software code libraries can be used. For example, thisfunctionality can be provided by the Microsoft HoloToolkit or the GoogleProject Tango platform. Various techniques have been described forspatial mapping and tracking including those described in U.S. Pat. No.9,582,717, which is expressly incorporated by reference herein.

Registration of Virtual Patient Data and Live Patient Data Using VisualAnatomic Features

-   a) Visual registration of virtual patient data in relationship to    live patient data by the surgeon or operator

In some embodiments, a surgeon or operator can visually align or matchvirtual patient data with live patient data. Such visually aligning ormatching of virtual patient data and live patient data can, for example,be performed by moving the OHMD, for example via movement of the head ofthe operator who is wearing the OHMD. In this example, the virtualpatient data can be displayed in a fixed manner, not changingperspective as the operator moves the OHMD. The operator will move theOHMD until the live patient data are aligned or superimposed onto thefixed projection of the virtual patient data. Once satisfactoryalignment, matching or superimposition of the live patient data with thevirtual patient data has been achieved, the surgeon can execute aregistration command, for example via a voice command or a keyboardcommand. The virtual patient data and the live patient data are nowregistered. At this point, upon completion of the registration, thevirtual patient data will move corresponding to the movement of theOHMD, for example as measured via the movement of an integrated IMU,image and field of view tracking, e.g. using anchor points in an imageor field of view using an image and/or video capture system, and/or anattached navigation system with optical or RF or other trackers, whichcan be attached to the patient, the surgical site, a bone or any othertissue of the patient, the surgeon, the surgeon's arm, the surgeon'shead or an OHMD worn by the surgeon.

Thus, once a satisfactory alignment or match has been achieved thesurgeon can execute a command indicating successful registration. Theregistration can include changes in at least one of position,orientation, and magnification of the virtual data and the live data inorder to achieve the alignment or match. Magnification applied to thevirtual data can be an indication of the distance from the OHMD or thesurgeon's head to the matched tissue. As a means of maximizing theaccuracy of the registration, the estimated distance between the OHMDand the target tissue or the skin surface or other reference tissue canbe confirmed with an optional physical measurement of the distance, inparticular if the OHMD is, for example, in a fixed position, e.g. on astand or tripod, which may be used optionally during the initialregistration. Upon successful alignment or matching, the surgeon commandcan register, for example, the virtual patient data and the live patientdata or images and the OHMD in the same common coordinate system.Virtual and physical surgical instruments and implant components canalso be registered in the common coordinate system.

In some embodiments of the invention, the visual anatomic data can be,for example, gyri of the brain or osteophytes or bone spurs orpathologic bone deformations or tumor nodes or nodules, e.g. on thesurface of a liver or a brain.

In some embodiments of the invention, the registration of virtualpatient data and live patient data using the methods described hereincan be repeated after one or more surgical steps have been performed. Inthis case, the surgically altered tissue or tissue surface or tissuecontour or shape, e.g. shape of a bone after milling or reaming, ortissue perimeter, e.g. perimeter of a bone cut, or tissue volume orother tissue features in the live patient can be matched to,superimposed onto and/or registered with the surgically altered tissueor tissue surface or tissue contour or tissue perimeter or tissue volumeor other tissue features in the virtual data of the patient, e.g. in avirtual surgical plan developed for the patient, with substantiallyidentical view angle of the virtual data of the patient seen by thesurgeon's eyes through the display of the OHMD unit and the live data ofthe patient seen by the surgeon's eyes through the OHMD unit. Thematching, superimposing and/or registering of the live data of thepatient and the virtual data of the patient after the surgical tissuealteration can be performed using the same methods described in theforegoing or any of the other registration methods described in thespecification or any other registration method known in the art.Referring to FIG. 3, FIG. 3 illustrates an example of registering adigital hologram or virtual data for an initial surgical step,performing the surgical step and re-registering one or more hologramsfor subsequent surgical steps. An optical head mounted display canproject or display a digital hologram of virtual data or virtual data ofthe patient 55. The digital hologram can optionally be fixed to the OHMDso that it will move with the movement of the OHMD 56. The operator canmove the OHMD until digital hologram of the virtual data or virtual dataof the patient is superimposed and aligned with the live data of thepatient, e.g. the surgical site 57. The digital hologram of the virtualdata or virtual data can then be registered using the same or similarcoordinates as those of the live data with which the digital hologram issuperimposed 58. The surgeon can then perform one or more predeterminedsurgical steps, e.g. bone cuts 59. A digital hologram of the virtualdata or virtual data can optionally be registered or re-registered afterthe surgical alteration with the live data 60. The digital hologram ofthe virtual data or virtual data after the surgical alteration canoptionally be displayed by the OHMD 61. The digital hologram of thevirtual data or virtual data after the surgical alteration canoptionally be fixed relative to the OHMD so that it will move with themovement of the OHMD 62. The operator can move the OHMD until digitalhologram of the virtual data or virtual data of the patient after thesurgical alteration is superimposed and aligned with the live data ofthe patient after the surgical alteration 63. The digital hologram ofthe virtual data or virtual data can then be registered using the sameor similar coordinates as those of the live data after the surgicalalteration with which the digital hologram is superimposed 64. Thesurgeon can then perform one or more predetermined subsequent surgicalsteps, e.g. bone cuts, milling or drilling 65. The preceding steps canoptionally be repeated until the surgical procedures is completed 66. Avirtual surgical plan 67 can be utilized. Optionally, the native anatomyof the patient including after a first surgical alteration can bedisplayed by the OHMD 68. The OHMD can optionally display digitalholograms of subsequent surgical steps 69.

-   b) Automatic or semi-automatic registration of virtual patient data    in relationship to live patient data using image processing and/or    pattern recognition and matching techniques-   c) In some embodiments of the invention, image processing    techniques, pattern recognition techniques or deep    learning/artificial neural-network based techniques can be used to    match virtual patient data and live patient data. Optionally, image    processing and/or pattern recognition algorithms can be used to    identify certain features, e.g. gyri or sulci on the brain surface    of virtual data of a patient. An ear including its unique shape can    also be used for the purpose of matching virtual patient data and    live patient data.

For example, with brain surgery, the patient can be placed on theoperating table. Optionally, cleaning or sterilization fluid can beapplied to the shaved skull, for example using betadine. The OHMD can beplaced over the patient, either on a tripod or worn by the operator, forexample with the head of the patient turned sideways over the livepatient's ear and lateral skull. The OHMD will be placed over an area ofthe live patient that includes the virtual data of the patient to bedisplayed.

Virtual data of the patient can be displayed in the OHMD. The virtualdata of the patient can include, for example, a visualization of thepatient's skin or other data, e.g. the patient's ear or nose, forexample derived from preoperative MRI data. The virtual data of thepatient's skin or other structures, e.g. the patient's ear or nose, canbe displayed simultaneous with the live patient data. The virtual dataof the patient can then be moved, re-oriented, re-aligned and,optionally, magnified or minified until a satisfactory alignment, matchor superimposition has been achieved. Optionally, the OHMD can be movedalso during this process, e.g. to achieve a satisfactory size matchbetween virtual data and live data of the patient, optionally withoutmagnification or minification of the virtual data of the patient.

Once a satisfactory alignment, match or superimposition has beenachieved between virtual data and live data of the patient, the operatorcan execute a command indicating successful registration. Changes inposition, orientation, or direction of the OHMD, for example as measuredvia an integrated IMU, image and field of view tracking, e.g. usinganchor points in an image or field of view using an image and/or videocapture system, and/or a navigation system attached to the OHMD, can beused to move the virtual patient data with the view of the live patientdata through the OHMD, with substantially identical object coordinatesof the virtual data of the patient and the live data of the patient,thereby maintaining registration during the course of the surgeryirrespective of any movements of the OHMD, e.g. head movement by theoperator wearing the OHMD, and ensuring that the virtual data of thepatient is correctly superimposed with the live data of the patient whenprojected into the surgeon's view.

After successful registration of the virtual patient data to thepatient's skin or other structures, e.g. an ear or a nose, the operatoror an assistant can apply a marker or calibration or registrationphantom or device on the patient, for example close to the intended siteof a craniotomy. The marker or calibration or registration phantom ordevice will not be covered by any drapes or surgical covers that will beplaced subsequently. A secondary registration of the virtual patientdata to the live patient data can then occur, by registering the virtualpatient data to the live patient data, using the live marker orcalibration or registration phantom or device placed on the patient andby cross-referencing these to the live data of the patient's skin orother structures, e.g. an ear or a nose. This can be achieved, forexample, by registering the patient's skin or other structures, e.g. anear or a nose, in the same coordinate system as the marker orcalibration or registration phantom or device placed on the patient,e.g. by co-registering the virtual patient data of the patient's skin orother structures, e.g. an ear or a nose or an osteophyte or bone spur orother bony anatomy or deformity, with the live data of the marker orcalibration or registration phantom or device. The distance, offset,angular offset or overall difference in coordinates between thepatient's skin or other structures, e.g. an ear or nose or an osteophyteor bone spur or other bony anatomy or deformity, to the marker orcalibration or registration phantom or device attached to the patientcan be measured and can be used to switch the registration of thevirtual patient data to the live patient data from the live data of thepatient's skin or other structures, e.g. an ear or a nose, to the livedata of the marker or calibration or registration phantom or device.Optionally, registration can be maintained to both the live data of thepatient's skin or other structures, e.g. an ear or a nose, and the livedata of the marker or calibration or registration phantom or device.Optionally, the system can evaluate if registration to the live data ofthe patient's skin or other structures, e.g. an ear or a nose, or to thelive data of the marker or calibration or registration phantom or deviceis more accurate and the system can switch back and forth betweeneither. For example, if the distance increases or decreases from theOHMD to the patient's skin or other structure, e.g. an ear or a nose,beyond a certain level, e.g. a threshold, which can be optionallypredefined, or if some of them is partially covered by a drape, thesystem can switch the registration to the live data of the marker orcalibration or registration phantom or device. The reverse is possible.Or, if the angle from the OHMD increases or decreases beyond a certainlevel, e.g. a threshold, which can be optionally predefined, to thepatient's skin or other structure, e.g. an ear or a nose or anosteophyte or bone spur or other bony anatomy or deformity, the systemcan switch the registration to the live data of the marker orcalibration or registration phantom or device. The reverse is possible.

The operator or the assistants can then place sterile drapes or surgicalcovers over the site, however preferably not covering the marker orcalibration or registration phantom or device. Registration can bemaintained via the live data of the marker or calibration orregistration phantom or device attached to the patient, e.g. adjacent toor inside a craniotomy site.

Image processing and/or pattern recognition of the live data of thepatient can then be performed through the OHMD, e.g. using a built inimage capture apparatus for capturing the live data of the patient orimage and/or video capture systems attached to, integrated with orcoupled to the OHMD.

Virtual and live data features or patterns can then be matched. Thematching can include a moving and/or reorienting and/or magnificationand/or minification of virtual data for successful registration with thelive data of the patient and superimposition of both. Virtual and livedata can include an osteophyte or bone spur or other bony anatomy ordeformity. Combination of (a) and (b), e.g. automatic registration withmanual adjustment option, e.g. by moving the virtual image data inrelation to the live image data after image processing software and/orpattern recognition software and/or matching software have identified apotential match or performed an initial matching, which can then befollowed by manual/operator based adjustments. Alternatively,manual/operator based matching and registration can be performed first,followed then by fine-tuning via software or algorithm (imageprocessing, pattern recognition, etc.) based matching and registration.Virtual and live data can include an osteophyte or bone spur or otherbony anatomy or deformity.

In some embodiments of the invention, the registration of virtualpatient data and live patient data using the methods described hereincan be repeated after one or more surgical steps have been performed. Inthis case, the surgically altered tissue or tissue surface or tissuecontour or tissue perimeter or tissue volume or other tissue features inthe live patient can be matched to, superimposed onto and/or registeredwith the surgically altered tissue or tissue surface or tissue contouror tissue perimeter or tissue volume or other tissue features in thevirtual data of the patient, e.g. in a virtual surgical plan developedfor the patient. The matching, superimposing and/or registering of thelive data of the patient and the virtual data of the patient after thesurgical tissue alteration can be performed using the same methodsdescribed in the foregoing or any of the other registration methodsdescribed in the specification or any other registration method known inthe art.

Registration of Virtual Patient Data and Live Patient Data UsingAnatomic Landmarks

In some embodiments, a surgeon can identify select anatomic landmarks onvirtual data of the patient, e.g. on an electronic preoperative plan ofthe patient, and on live data of the patient. For example, the surgeoncan identify a landmark by placing a cursor or a marker on it on anelectronic image of the virtual data of the patient and by clicking onthe landmark once the cursor or marker is in the desired location. In aspine, such a landmark can be, for example, the posterior tip of aspinous process, a spinal lamina, an inferior facet on the patient'sleft side, a superior facet on the patient's left side, an inferiorfacet on the patient's right side, a superior facet on the patient'sright side, a tip of a facet joint, a bone spur, an osteophyte etc. In ahip, such landmarks can be the most anterior point of the acetabulum, anosteophyte, e.g. on the acetabular rim, in the acetabulum, adjacent tothe acetabulum, on the femoral head, on the femoral neck or the neckshaft junction, the center of the femoral head in a 2D or 3D image, themost anterior point of the femoral head, an anterosuperior iliac spine,an anteroinferior iliac spine, a symphysis pubis, a greater trochanter,a lesser trochanter etc. In a knee, such landmarks can be a femoralcondyle, a femoral notch, an intercondylar space, a medial or lateralepicondyle, a femoral axis, an epicondylar axis, a trochlear axis, amechanical axis, a trochlear groove, a femoral osteophyte, a marginalfemoral osteophyte, a central femoral osteophyte, a dome of the patella,a superior, medial, lateral, inferior edge of the patella or the femuror femoral articular surface, a patellar osteophyte, an anterior tibia,a tibial spine, a medial, lateral, anterior, posterior edge of thetibia, a tibial osteophyte, a marginal tibial osteophyte, a centraltibial osteophyte. The surgeon can then identify the same landmarks livein the patient. For example, as the surgeon looks through the OHMD, thesurgeon can point with the finger or with a pointing device at thecorresponding anatomic landmark in the live data. The tip of the pointeror the tip of the finger can, optionally, include a tracker whichlocates the tip of the pointer or the finger in space. Such locating canalso be done visually using image capture, e.g. in a stereoscopic mannerthrough the OHMD for more accurate determination of the distance andlocation of the pointer or finger in relationship to the OHMD. An imageand/or video capture systems can also be attached to, integrated with orcoupled to the OHMD. Virtual and live data can include an osteophyte orbone spur or other bony anatomy or deformity.

Representative anatomic landmarks that can be used for registration ofvirtual and live data of the patient can include (but are not limitedto):

In Spine:

-   -   A portion or an entire spinous process    -   A portion or an entire spinal lamina    -   A portion or an entire spinal articular process    -   A portion of or an entire facet joint    -   A portion of or an entire transverse process    -   A portion of or an entire pedicle    -   A portion of or an entire vertebral body    -   A portion of or an entire intervertebral disk    -   A portion of or an entire spinal osteophyte    -   A portion of or an entire spinal bone spur    -   A portion of or an entire spinal fracture    -   A portion of or an entire vertebral body fracture    -   Combinations of any of the foregoing

Hip:

-   -   A portion of or an entire acetabulum    -   A portion of or an entire edge of an acetabulum    -   Multiple portions of an edge of an acetabulum    -   A portion of an iliac wall    -   A portion of a pubic bone    -   A portion of an ischial bone    -   An anterior superior iliac spine    -   An anterior inferior iliac spine    -   A symphysis pubis    -   A portion of or an entire greater trochanter    -   A portion of or an entire lesser trochanter    -   A portion of or an entire femoral shaft    -   A portion of or an entire femoral neck    -   A portion of or an entire femoral head    -   A fovea capitis    -   A transverse acetabular ligament    -   A pulvinar    -   A ligamentum teres    -   A labrum    -   One or more osteophytes, femoral and/or acetabular    -   Combinations of any of the foregoing

Knee:

-   -   A portion or an entire medial femoral condyle    -   A portion or an entire lateral femoral condyle    -   A portion or an entire femoral notch    -   A portion or an entire trochlea    -   A portion of an anterior cortex of the femur    -   A portion of an anterior cortex of the femur with adjacent        portions of the trochlea    -   A portion of an anterior cortex of the femur with adjacent        portions of the trochlea and osteophytes when present    -   One or more osteophytes femoral and/or tibial    -   One or more bone spurs femoral and/or tibial    -   An epicondylar eminence    -   A portion or an entire medial tibial plateau    -   A portion or an entire lateral tibial plateau    -   A portion or an entire medial tibial spine    -   A portion or an entire lateral tibial spine    -   A portion of an anterior cortex of the tibia    -   A portion of an anterior cortex of the tibia and a portion of a        tibial plateau, medially or laterally or both    -   A portion of an anterior cortex of the tibia and a portion of a        tibial plateau, medially or laterally or both and osteophytes        when present    -   A portion or an entire patella    -   A medial edge of a patella    -   A lateral edge of a patella    -   A superior pole of a patella    -   An inferior pole of a patella    -   A patellar osteophyte    -   An anterior cruciate ligament    -   A posterior cruciate ligament    -   A medial collateral ligament    -   A lateral collateral ligament    -   A portion or an entire medial meniscus    -   A portion or an entire lateral meniscus    -   Combinations of any of the foregoing

Shoulder:

-   -   A portion or an entire glenoid    -   A portion or an entire coracoid process    -   A portion or an entire acromion    -   A portion of a clavicle    -   A portion or an entire humeral head    -   A portion or an entire humeral neck    -   A portion of a humeral shaft    -   One or more humeral osteophytes    -   One or more glenoid osteophytes    -   A portion or an entire glenoid labrum    -   A portion or an entire shoulder ligament, e.g. a coracoacromial        ligament, a superior, middle, or inferior glenohumeral ligament    -   A portion of a shoulder capsule    -   Combinations of any of the foregoing

Skull and Brain:

-   -   A portion of a calvarium    -   A portion of an occiput    -   A portion of a temporal bone    -   A portion of a occipital bone    -   A portion of a parietal bone    -   A portion of a frontal bone    -   A portion of a facial bone    -   A portion of a facial structure    -   A portion or an entire bony structure inside the skull    -   Portions or all of select gyri    -   Portions or all of select sulci    -   A portion of a sinus    -   A portion of a venous sinus    -   A portion of a vessel    -   A portion of an ear    -   A portion of an outer auditory canal

Organs:

-   -   A portion of an organ, e.g. a superior pole or inferior pole of        a kidney    -   An edge or a margin of a liver, a spleen, a lung    -   A portion of a hepatic lobe    -   A portion of a vessel    -   A portion of a hiatus, e.g. in the liver or spleen    -   A portion of a uterus

Someone skilled in the art can identify other anatomic landmarks of hardtissues, soft-tissues and or organs including brain that can be used forregistration of virtual data (including optionally including virtualsurgical plans) and live data of the patient and the OHMD in a commoncoordinate system. Virtual and physical surgical instruments and implantcomponents can also be registered in the common coordinate system.

In some embodiments of the invention, the OHMD can display an arbitraryvirtual plane over the surgical field. The arbitrary virtual plane canbe moveable using a virtual or other interface. For example, thearbitrary virtual plane can include a “touch area”, wherein gesturerecognition software, for example the one provided by Microsoft with theMicrosoft Hololens including, for example, the integrated virtual “dragfunction” for holograms can be used to move the arbitrary virtual plane.For example, one or more cameras integrated or attached to the OHMD cancapture the movement of the surgeon's finger(s) in relationship to thetouch area; using gesture tracking software, the virtual plane can thenbe moved by advancing the finger towards the touch area in a desireddirection.

The OHMD can display the arbitrary virtual plane in any locationinitially, e.g. projected onto or outside the surgical field, e.g. a hipjoint, knee joint, shoulder joint, ankle joint, or a spine. The OHMD canoptionally display the arbitrary virtual plane at a defined angle, e.g.orthogonal or parallel, relative to a fixed structure in the operatingroom, which can, for example, be recognized using one or more cameras,image capture or video capture systems integrated into the OHMD andspatial recognition software such as the one provided by Microsoft withthe Microsoft Hololens or which can be recognized using one or moreattached optical markers or navigation markers including infrared or RFmarkers. For example, one or more optical markers can be attached to anextension of the operating table. The OHMD can detect these one or moreoptical markers and determine their coordinates and, with that, thehorizontal plane of the operating room table. The arbitrary virtualplane can then be displayed perpendicular or at another angle relativeto the operating room table.

For example, in a hip replacement, the OHMD can display a virtualarbitrary plane over the surgical site. The virtual arbitrary plane canbe perpendicular to the operating table. Using a virtual interface, e.g.a touch area on the virtual surgical plane and gesture tracking, theOHMD can detect how the surgeon is moving the virtual arbitrary plane.Optionally, the virtual arbitrary plane can maintain its perpendicular(or of desired other angle) orientation relative to the OR table whilethe surgeon is moving and/or re-orienting the plane; a perpendicularorientation can be desirable when the surgeon intends to make aperpendicular femoral neck cut. A different angle can be desirable, whenthe surgeon intends to make the femoral neck cut with anotherorientation.

Using the touch area or other virtual interface, the surgeon can thenmove the arbitrary virtual plane into a desired position, orientationand/or alignment. The moving of the arbitrary virtual plane can includetranslation and rotation or combinations thereof in any desireddirection using any desired angle or vector. The surgeon can move thearbitrary virtual plane to intersect with select anatomic landmarks orto intersect with select anatomic or biomechanical axes. The surgeon canmove the arbitrary virtual plane to be tangent with select anatomiclandmarks or select anatomic or biomechanical axes.

For example, in a hip replacement, the surgeon can move the arbitraryvirtual plane to be tangent with the most superior aspect of the greatertrochanter and the most superior aspect of the lesser trochanter. FIG.4A shows an illustrative example of a virtual plane 70 that a primarysurgeon has moved and aligned to be tangent with the most superioraspect of the greater trochanter 71 and the most superior aspect of thelesser trochanter 72. FIG. 4B shows an illustrative example of the samevirtual plane 70 that the primary surgeon has moved and aligned to betangent with the most superior aspect of the greater trochanter 71 andthe most superior aspect of the lesser trochanter 72, now with the viewfrom the optical head mounted display of a second surgeon or surgicalassistant, e.g. on the other side of the OR table.

Optionally, for example with a pointer with an attached optical markeror an attached navigation marker, or with his finger detected using animage or video capture system integrated into the OHMD and gesturerecognition software such as the one provided by Microsoft with theHololens, or with his finger with an attached optical marker ornavigation marker, the surgeon can point at and identify the sulcuspoint, e.g. the lowest point between the greater trochanter and thefemoral neck, which can be an additional reference. The line connectingthe most superior aspect of the greater trochanter and the most superioraspect of the lesser trochanter can then be determined on apre-operative or intra-operative AP radiograph of the hip; optionally,the sulcus point can also be detected on the AP radiograph. The APradiograph can include a template used by the surgeon for selecting andsizing, for example, the femoral and acetabular component, as well asthe liner and/or femoral heads. The radiographic template can include anindication for the femoral neck cut. The angle between the lineconnecting the most superior aspect of the greater trochanter and themost superior aspect of the lesser trochanter and the indication for thefemoral neck cut can be determined. FIG. 4C is an illustrative examplethat shows that a second virtual plane 73, the virtual femoral neck cutplane 73, can then be projected or displayed by the OHMD, alsoperpendicular to the OR table like the arbitrary virtual plane 70, thelatter tangent with the most superior aspect of the greater trochanter71 and the most superior aspect of the lesser trochanter 72, and thefemoral neck cut plane 73 at the same angle and/or distance to thearbitrary virtual plane as the angle and distance between the lineconnecting the most superior aspect of the greater trochanter and themost superior aspect of the lesser trochanter and the indication for thefemoral neck cut on the radiograph. In this manner, the femoral neck cutplane can be defined using a second virtual plane prescribed orpredetermined based on the intra-operatively placed arbitrary virtualplane, moved by the operator to be tangent with the most superior aspectof the greater trochanter and the most superior aspect of the lessertrochanter. The virtual femoral neck cut plane prescribed and projectedor displayed in this manner can also be a virtual guide, e.g. a virtualcut block that projects, for example, a virtual slot for guiding anphysical saw. The virtual guide or virtual cut block can have one ormore dimensions identical to an physical guide or cut block, so that thephysical guide or cut block can be aligned with the virtual guide or cutblock. The virtual guide or cut block can be an outline, 2D or 3D,partial or complete, of the physical guide or cut block, with one ormore identical dimensions, so that the surgeon can align the physicalguide or cut block with the virtual guide or cut block. The virtualguide or cut block can include placement indicia for the physical guideor cut block.

If radiographic magnification is a concern for prescribing a secondvirtual plane, e.g. a virtual cut plane, based on a first virtual plane,e.g. a plane tangent with or intersecting one or more anatomic landmarksor one or more anatomic or biomechanical axes, at an angle incorporatedfrom or derived from a pre-operative radiograph, optionally, distancemeasurements can be incorporated and magnification correction can beapplied. For example, the distance between one or more landmarks, e.g.the ones with which the virtual plane is tangent with or that thevirtual plane intersects, can be measured in the live data of thepatient and can be measured on the radiograph. If the radiographicdistance is larger or smaller than the distance in the live patient, amagnification correction can be applied and, for example, the distancebetween the first virtual plane, e.g. a plane tangent with orintersecting one or more anatomic landmarks or one or more anatomic orbiomechanical axes, and the second virtual plane, e.g. a virtual cutplane, can be corrected based on the radiographic magnification factor.

In another example, an arbitrary virtual plane can be projected ordisplayed outside of or over the surgical field in a knee replacement.Optionally, the arbitrary virtual plane can be, at least initially,perpendicular to the OR table or at a defined angle to the OR table. Ifthe mechanical axis of the leg has been determined in a preceding step,e.g. using an intra-operative measurement, for example with opticalmarkers applied to the thigh and one or more optical markers applied tothe ankle joint, for determining the center of rotation of the hip jointand the center of the ankle joint using an image capture or videocapture system integrated into, attached to or separate from the OHMD,the arbitrary virtual plane can be configured to be perpendicular to themechanical axis of the leg. Using a virtual interface, e.g. a toucharea, and an image or video capture system integrated or attached to theOHMD and optional gesture tracking software, the surgeon can move and/orre-align the arbitrary virtual plane, for example to intersect with themedial and lateral joint space of the exposed knee joint, for example inextension or at 5, 10, 15, 20, 30, 45, or more degrees of flexion. FIG.5 is an illustrative example of an arbitrary virtual plane 74 in theknee that intersects with the medial 76 and lateral 75 joint space inextension.

One or more additional arbitrary virtual planes can then optionally beprojected, for example perpendicular or at another angle relative to theoperating table or using a desired femoral component flexion angle or adesired tibial slope. The surgeon can optionally move these one or morearbitrary virtual planes to coincide with one or more anatomic axes, forexample the anatomic femoral shaft axis or the anatomic tibial shaftaxis in the live patient. The surgeon can also move a virtual arbitraryplane to be placed and oriented in the center of the femoral notch,parallel to the notch walls and extending centered between the medialand the lateral femoral shaft cortex as a means of estimating theanatomic femoral shaft axis.

Once the anatomic femoral and/or tibial axes have been determined orestimated, a virtual surgical plan with femoral and tibial resectionsdesigned to achieve a desired femoral mechanical axis correction, e.g.from the patient's mechanical axis alignment, e.g. 5, 10, 15 degrees ofvarus or valgus, to normal mechanical axis alignment or any desiredresidual, e.g. congenital varus or valgus, can be developed orgenerated. Implant size and desired polyethylene thickness can befactored into the virtual surgical plan. The OHMD can then, for example,project virtual surgical cut planes based on the virtual surgical planand/or the intra-operative measurements, the desired varus and/or valguscorrection, desired slope, and/or desired implant rotation. The surgeoncan then align the physical saw blade with the projected or displayedvirtual saw blade. Alternatively, the OHMD can display a virtual guideor virtual cut block with at least one or more dimensions identical tothe physical guide or physical cut block and the surgeon can align thephysical cut guide or cut block with the virtual guide or cut block, inthe physical guide or cut block, insert the saw blade into the physicalguide or cut block and execute the one or more blocks.

The foregoing concepts of projecting arbitrary virtual planes andaligning them with one or more anatomic landmarks, anatomic axes orbiomechanical or mechanical axes can be applied to any joint and alsothe spine. Similarly, these concepts can be applied to brain surgery,where one or more virtual planes can be projected or displayed and movedto be tangent with or intercept one or more landmarks, e.g. gyri, pons,cerebellum etc. Similarly, these concepts can be applied to organsurgery, where one or more virtual planes can be projected or displayedand moved to be tangent with or intercept one or more landmarks, e.g.liver portal, anterior liver edge, one or more cardiac valves etc.

Other arbitrary 2D and/or 3D virtual shapes or outlines or surfaces,e.g. cubes, cuboids, prisms, cones, cylinders, spheres, ellipsoidderived 3D shapes, irregular shapes, can be virtually projected ordisplayed and automatically or using a virtual or other user interfacemoved, oriented or aligned to coincide, to be tangent with, tointersect, to be partially or completely superimposed with patientanatomy, pathology, anatomic axes, biomechanical including mechanicalaxes, anatomic planes, 3D shapes, 2D and/or 3D geometries, 3D surfaces,and/or 3D volumes of any internal organs, soft-tissues or hard tissuesof the patient; after the moving, orienting or aligning, the coordinateinformation of the 2D and/or 3D virtual shapes or outlines or surfacescan then be measured. Optionally, based on the coordinate information,additional intraoperative measurements can be performed and/or,optionally, a virtual surgical plan can be developed or modified usingthe information.

In some embodiments of the invention, the registration of virtualpatient data and live patient data using the methods described hereinincluding anatomic landmarks can be repeated after one or more surgicalsteps have been performed. In this case, the surgically altered tissueor tissue surface or tissue contour or tissue perimeter or tissue volumeor other tissue features in the live patient can be matched to,superimposed onto and/or registered with the surgically altered tissueor tissue surface or tissue contour or tissue perimeter or tissue volumeor other tissue features in the virtual data of the patient, e.g. in avirtual surgical plan developed for the patient. The matching,superimposing and/or registering of the live data of the patient and thevirtual data of the patient after the surgical tissue alteration can beperformed using the same methods described in the foregoing or any ofthe other registration methods described in the specification or anyother registration method known in the art. Optionally, differentanatomic landmarks can also be used for the first registration and anyof the subsequent registrations. Or the same anatomic landmarks can beused for the first registration and any of the subsequent registrations.

Using Light Sources for Referencing Live Anatomic Landmarks

The tracker or pointing device can also be a light source, which can,for example, create a red point or green point created by a laser on thepatient's tissue highlighting the anatomic landmark intended to be usedfor registration. A light source can be chosen that has an intensityand/or a color that will readily distinguish it from the live tissue ofthe patient.

The laser or other light source can optionally be integrated into orattached to the OHMD. For example, the laser or the light source can beintegrated into or attached to a bridge connecting the frame piecesbetween the left and the right eye portion of the OHMD, for example overthe nasal region.

Image capture, for example integrated into or attached to or coupled tothe OHMD, can be used to identify the location of the light on thepatient's tissue or the patient's anatomic landmark. Once the light hasbeen directed to the desired location on the live data of the patient,specifically, the live landmark of the patient, registration can beperformed by executing a registration command, registering the live dataof the patient with the virtual data of the patient, e.g. the livelandmark with the laser or other light being reflected of it and thecorresponding virtual landmark of the patient. This process can berepeated for different anatomic landmarks, e.g. by pointing the lightsource at the next live anatomic landmark of the patient, confirmingaccurate placement or pointing, the light, e.g. a red or green laserpoint being reflected from the live patient landmark can be captured viathe image capture device and software of the OHMD, and the next anatomiclive landmark can be registered with the corresponding virtual anatomiclandmark of the patient. Virtual and live data can include an osteophyteor bone spur or other bony anatomy or deformity. In this manner, theOHMD, live data of the patient and virtual data of the patient can beregistered in a common coordinate system. Virtual and physical surgicalinstruments and implant components can also be registered in the commoncoordinate system.

In some embodiments, more than one live and virtual anatomic landmark ofthe patient will be used, e.g. two, three or more.

In some embodiments of the invention, ultrasound or a radiofrequencytransmitter can be used to pinpoint certain live anatomic landmarks. Forexample, an ultrasonic transmitter or a radiofrequency transmitter canbe integrated into a point device, for example the tip of a pointingdevice. When the tip touches the desired live anatomic landmark, thetransmitter can transmit and ultrasonic or RF signal which can becaptured at a receiving site, optionally integrated into the OHMD.Optionally, for example as a means of increasing the accuracy of livedata registration, multiple receiving sites can be used in spatiallydifferent locations. Virtual and live data can include an osteophyte orbone spur or other bony anatomy or deformity.

In some embodiments of the invention, the dimensions of the pointer havebeen previously scanned and registered with the OHMD. The image and/orvideo capture system attached to, integrated with or coupled to the OHMDcan recognize the pointer in the live data and can identify the tip ofthe pointer. When the tip of the pointer touches the live landmark onthe patient that corresponds to the landmark in the virtual data, thesurgeon can, for example, click to indicate successfulcross-referencing. The two data points can then optionally be fused orsuperimposed in a common coordinate system. Virtual and live data anddata points can include or can be generated from an osteophyte or bonespur or other bony anatomy or deformity. Virtual and physical surgicalinstruments and implant components can also be registered in the commoncoordinate system.

Anatomic landmarks can include an unalterated surface shape, e.g. skin,facial features, e.g. the tip of the nose, a distance between both eyes,the location of an ear, the shape of the ear. Anatomic landmarks canalso be bony landmarks, e.g. a medial or lateral malleolus, a tibialtuberosity, a medial or lateral epicondyle, a trochlear notch, a spinousprocess etc. Virtual and live data and virtual and live anatomiclandmarks can include an osteophyte or bone spur or other bony anatomyor deformity.

Optionally, a live anatomic surface can be used for registrationpurposes. In this embodiment, the live anatomic surface can be derived,for example, using a light scanning, infrared scanning or ultrasoundtechnique, or ultrasonic scanning technique during the surgery. The livesurfaces of the patient that are detected and generated in this mannercan be matched or aligned with virtual surfaces of the patient, forexample obtained preoperatively using an imaging test such as x-rayimaging, ultrasound, CT or MRI or any other technique known in the art.Virtual and live data and anatomic surfaces can include an osteophyte orbone spur or other bony anatomy or deformity.

In some embodiments of the invention, the registration of virtualpatient data and live patient data using the methods described hereincan be repeated after one or more surgical steps have been performed. Inthis case, the surgically altered tissue or tissue surface or tissuecontour or tissue perimeter or tissue volume or other tissue features inthe live patient can be matched to, superimposed onto and/or registeredwith the surgically altered tissue or tissue surface or tissue contouror tissue perimeter or tissue volume or other tissue features in thevirtual data of the patient, e.g. in a virtual surgical plan developedfor the patient. The matching, superimposing and/or registering of thelive data of the patient and the virtual data of the patient after thesurgical tissue alteration can be performed using the same methodsdescribed in the foregoing or any of the other registration methodsdescribed in the specification or any other registration method known inthe art.

Registration of Virtual Patient Data and Live Patient Data UsingImplantable or Attachable Markers or Calibration or RegistrationPhantoms or Devices Including Optical Markers

In some embodiments of the invention, a surgeon is optionally usingimplantable or attachable markers to register virtual data of thepatient with live data of the patient. This embodiment can, for example,be useful if the surgery is very extensive and results in the removal oftissue in the surgical site, as can be the case during brain surgery,e.g. removal of a brain tumor, liver surgery, e.g. removal of a livertumor, joint replacement surgery and many other types of surgery.Virtual and live data can include an osteophyte or bone spur or otherbony anatomy or deformity.

The terms implantable markers, attachable markers, skin markers,soft-tissue markers, calibration or registration phantoms or devices,and image capture markers as used throughout the application can includeoptical markers, e.g. optical markers with different geometric shapes orpatterns, with QR codes, with bar codes, with alphanumeric codes.Implantable or attachable markers or calibration or registrationphantoms or devices can be implanted prior to the actual surgery and canbe included in pre-, intra- and/or postoperative imaging. Implantable orattachable markers or calibration or registration phantoms or devicescan be implanted on or attached to osteophytes or bone spurs or otherbony anatomy or deformity.

If the implantable or attachable markers or calibration or registrationphantoms or devices are present in the virtual image data, the surgeoncan optionally identify the implantable or attachable markers orcalibration or registration phantoms or devices after an incision as heor she gains access to the target tissue and the implantable markersplaced next to the target tissue or inside the target tissue. Suchimplantable or attachable markers or calibration or registrationphantoms or devices can, for example, include radiation beets ormetallic beets, for example also used for stereographic imaging orregistration.

Alternatively, implantable or attachable markers or calibration orregistration phantoms or devices can be placed during the surgery and,for example using image capture through the OHMD or attached to,integrated with or coupled to the OHMD, the location of the implantableor attachable markers or calibration or registration phantoms or devicescan be determined. The location of the implantable or attachable markersor calibration or registration phantoms or devices on the patient in thelive data of the patient can then be matched with the location of theanatomic structure to which the implantable or attachable markers orcalibration or registration phantoms or devices is attached in thevirtual data of the patient. For example, the anatomic structure in thevirtual and live data can include an osteophyte or bone spur or otherbony anatomy or deformity. In some embodiments, a pointer or pointingdevice can optionally be placed on the implantable or attachable markersor calibration or registration phantoms or devices followed by imagecapture through the OHMD or other image capture device attached to,integrated with or coupled to the OHMD and registration of the tip ofthe pointer. In this manner, the OHMD, the implantable or attachablemarkers or calibration or registration phantoms or devices includingoptical markers and, through the use of the implantable or attachablemarkers or calibration or registration phantoms or devices includingoptical markers, the anatomic structures, pathologic structures,instruments, implant components and any other objects to which one ormore implantable or attachable markers or calibration or registrationphantoms or devices including optical markers can be attached, as wellas the virtual data of the patient can be registered in a commoncoordinate system. Virtual and physical surgical instruments and implantcomponents can also be registered in the common coordinate system.Implantable or attachable markers or calibration or registrationphantoms or devices can include rigid or fixed registration markers.Such rigid or fixed registration markers can be used to maintainregistration as surgical field is being altered. A rigid or fixedregistration marker can, for example, be a screw or a pin. Virtual andlive data can include an osteophyte or bone spur or other bony anatomyor deformity. The rigid or fixed registration marker can be attached tothe osteophyte or bone spur or other bony anatomy or deformity. In someembodiments, the medical device that is being implanted or a componentthereof that has been, for example, already temporarily or permanentlyattached to the patient's tissue, e.g. an osteophyte or bone spur orbony anatomy or deformity, or the anatomic site or the surgical site canbe used as an implantable or attachable marker or calibration orregistration phantom or device during the surgery, for example whilesubsequent steps of the surgery are being completed. Such subsequentsteps can, for example, include the implantation of additionalcomponents of the medical device. For example, in spinal fusion surgery,a first pedicle screw can be implanted. Live data and virtual data ofthe first pedicle screw can be registered. Subsequent pedicle screws orother components can be virtually displayed in the OHMD including theirintended path, position, location or orientation, by maintainingregistration between live and virtual data using the registered firstpedicle screw. Any other rigid or fixed registration marker orimplantable device can be used in this manner for different types ofsurgeries of the human body.

The one or more implantable or attachable markers or calibration orregistration phantoms or devices can be attached to bone, cartilage,soft-tissues, organs or pathologic tissues such as osteophytes or bonespur or other bony anatomy or deformity. etc.

The one or more implantable or attachable markers or calibration orregistration phantoms or devices can optionally include optical markers,retroreflective markers, infrared markers, or RF markers or any othermarker device described in the art.

Optical markers are markers that reflect light within the visiblespectrum, i.e. the portion of the electromagnetic spectrum that isvisible to the human eye, with wavelengths from about 390 to 700 nm or afrequency band from about 430-770 THz. Optical markers can also reflectlight that includes a mix of different wavelengths within the visiblespectrum. The light reflected by the optical markers can be detected byan image and/or video capture system integrated into, attached to orseparate from the OHMD. Optical markers can be detected with regard totheir location, position, orientation, alignment and/or direction ofmovement with use of an image and/or video capture system integratedinto, attached to or separate from the OHMD with associated imageprocessing and, optionally, pattern recognition software and systems.Optical markers can include markers with select geometric patternsand/or geometric shapes that an image and/or video capture system, forexample integrated into, attached to or separate from the OHMD, canrecognize, for example using image processing and/or pattern recognitiontechniques. Optical markers can include markers with select alphabeticcodes or patterns and/or numeric codes or patterns and/or alphanumericcodes or patterns or other codes or patterns, e.g. bar codes or QRcodes, that an image and/or video capture system, for example integratedinto, attached to or separate from the OHMD, can recognize, for exampleusing image processing and/or pattern recognition techniques. QR codesor quick response codes include any current or future generation matrixcode including barcode. Barcodes and QR codes are machine readableoptical labels that can include information, for example, about thepatient including patient identifiers, patient condition, type ofsurgery, about the surgical site, the spinal level operated if spinesurgery is contemplated, the patient's side operated, one or moresurgical instruments, one or more trial implants, one or more implantcomponents, including type of implant used and/or implant size, type ofpolyethylene, type of acetabular liner (e.g. standard, lipped, offset,other) if hip replacement is contemplated. A QR code can use differentstandardized encoding modes, e.g. numeric, alphanumeric, byte/binary,and/or kanji to store data. Other encoding modes can be used. Anycurrent and/or future version of QR codes can be used. QR codes usingsingle or multi-color encoding can be used. Other graphical markers,such as the ones supported by the Vuforia (PTC, Needham, Mass.)augmented reality platform, can be used as well.

A bar code, QR code or other graphical marker can be the optical marker.A bar code, QR code or other graphical marker can be part of an opticalmarker or can be integrated into an optical marker. The same QR code orbar code or other graphical marker can contain

-   -   information related to the patient and/or the surgical site,        e.g. patient identifiers, age, sex, BMI, medical history, risk        factors, allergies, site and side (left, right), spinal level to        be operated    -   information related to inventory management, e.g. of surgical        instruments and/or implants or implant components, e.g. left vs.        right component, selected component size (match against virtual        surgical plan and/or templating and/or sizing)

and can be used to obtain information about the location position,orientation, alignment and/or direction of movement, if applicable, ofthe surgical site, surgically altered tissue, one or more surgicalinstruments and one or more trial implants and/or implant components.

Geometric patterns, geometric shapes, alphabetic, numeric, alphanumeric,and other codes or patterns including bar codes and QR codes included inor part of one or more optical markers can be predefined and,optionally, stored in database accessible by an image and/or videocapture system and associated image processing software and patternrecognition software. Geometric patterns, geometric shapes, alphabetic,numeric, alphanumeric, and other codes or patterns including bar codesand QR codes included in or part of one or more optical markers can bein 2D and some of it in 3D. For example, one or more planar or 2Dpatterns can be used in select embodiments. Alternatively, select 3Dgeometric shapes can be used, e.g. cubes, cuboids, prisms, cones,cylinders, spheres. Any 3D shape can be used including irregular shapesand/or asymmetric shapes. The 3D geometric shape can include 2Dgeometric patterns and/or alphabetic, numeric, alphanumeric, and othercodes or patterns including bar codes and QR codes on one or moresurfaces. For example, if a cuboid is used, the same or differentgeometric patterns and/or alphabetic, numeric, alphanumeric, and othercodes or patterns including bar codes and QR codes can be included in,affixed to or integrated into one or more of its surfaces or faces, e.g.two opposing surfaces or two adjacent surfaces oriented, for example,perpendicularly. 2D geometric patterns and/or alphabetic, numeric,alphanumeric, and other codes or patterns including bar codes and QRcodes included in, affixed to or integrated into one or more surfaces orfaces of a 3D geometric shape can be used to determine the orientationof select surfaces or faces of the geometric shape including the opticalmarker and, with that, the orientation and/or alignment of the surfaceor face and with that the geometric shape, for example in relationshipto a surgical site, a surgical alteration, e.g. a cut bone surface or areamed bone surface, a surgical instrument and/or one or more implantcomponents including trial implants.

Geometric patterns and/or geometric shapes, alphabetic, numeric,alphanumeric, and other codes or patterns including bar codes and QRcodes can be in color or black and white. Geometric patterns and/orgeometric shapes and/or alphabetic, numeric, alphanumeric, and othercodes or patterns including bar codes and QR codes can include portionsthat include color and black and white sections, portions that includeonly color and portions that are only black and white. Geometric shapescan include faces or surfaces that include color and black and white,faces or surfaces that include only black and white, and faces orsurfaces that include only color. Different colors and different colorcodes can be used for different faces or surfaces of a geometric shapepart of an optical marker. Different colors and different color codescan be used for different geometric patterns and/or geometric shapesand/or alphabetic, numeric, alphanumeric, and other codes or patternsincluding bar codes and QR codes. Different colors and different colorcodes can be used for different optical markers. Different colors, e.g.red, blue, green, orange, cyan etc., can be used for different geometricpatterns and/or geometric shapes and/or alphabetic, numeric,alphanumeric, and other codes or patterns including bar codes and QRcodes. Different colors, e.g. red, blue, green, orange, yellow, pink,cyan can be used for different optical markers. Different opticalmarkers can optionally be associated with different surgical stepsand/or different surgical instruments and/or different implantcomponents; the use of a particular marker can be recognized by an imageand/or video capture system integrated into, attached to or separatefrom the OHMD using standard image processing and/or pattern recognitionsoftware, including, optionally a database of patterns, e.g. with theirassociations with a particular surgical step and/or surgicalinstruments. As the image and/or video capture system recognizes aparticular optical marker in the field of view, for example based on aparticular geometric patterns and/or geometric shape and/or alphabetic,numeric, alphanumeric, and other codes or patterns including bar codesand QR codes used, it can then optionally display the correspondingsurgical step and/or surgical instrument and/or implant componentassociated with that optical marker.

2D geometric patterns, alphabetic, numeric, alphanumeric, and othercodes or patterns including bar codes and QR codes or combinationsthereof, optionally with color and/or black and white coding, includedin, affixed to or integrated into one or more surfaces or faces of a 3Dgeometric shape can be used to determine the orientation and/oralignment of select surfaces or faces of the geometric shape and, withthat, the orientation and/or alignment of the geometric shape and/or theoptical marker, for example in relationship to an anatomic landmark, asurgical site, a surgical alternation, e.g. a cut bone surface or areamed bone surface, a surgical instrument and/or one or more implantcomponents including trial implants. One or more 2D geometric patterns,alphabetic, numeric, alphanumeric, and other codes or patterns includingbar codes and QR codes, optionally with color and/or black and whitecoding, included in, affixed to or integrated into an optical marker canbe used to determine the orientation and/or alignment of the opticalmarker, which can, for example, be affixed to or integrated into ananatomic landmark, a surgical site, a surgical alternation, e.g. a cutbone surface or a reamed bone surface, a surgical instrument and/or oneor more implant components including trial implants. Optical markers canbe affixed to an anatomic landmark, a surgical site, a surgicalalteration, e.g. a cut bone surface or a reamed bone surface, or a drillhole of the patient and the corresponding anatomic landmark, surgicalsite, or surgical alternation can be identified in the virtual data ofpatient thereby enabling registration of the virtual data and the livedata of the patient in the same coordinate system. Optical markers canalso be attached to an OHMD including multiple OHMDs if multiple OHMDsare used during a surgery. Optionally, optical markers, e.g. with QRcodes, can be used to differentiate a first from a second, third, fourthand/or more OHMDs. One or more optical markers can optionally beattached to the operating room table and they can be registered in acoordinate system, for example the same coordinate system in which theone or more OHMDs, the patient, and portions of the surgical site can beregistered. One or more optical markers can optionally be attached toother structures in the operating room including fixed structures, e.g.walls, and movable structures, e.g. OR lights, and they can beregistered in a coordinate system, for example the same coordinatesystem in which the one or more OHMDs, the patient, and portions of thesurgical site can be registered. In this example, optical markers canalso be mounted to fixed structures on holding arms or extenders,optionally moveable and, for example, of known dimensions, orientations,lengths and angles.

Optical markers attached to fixed structures such as OR walls can beused to enhance the accuracy of room recognition and spatial mapping, inparticular when the coordinates and/or the angles and/or distancesbetween different optical markers are known. Optical markers attached tofixed structures such as OR walls can also be used to enhance thedetermination of the location and pose and change in location or pose orthe coordinates and change in coordinates of one or more optical headmounted displays, which can assist with increasing the accuracy of thedisplay of virtual data and their superimposition on corresponding livedata. Optical markers attached to movable structures can be used totrack their location in the operating room. Optical markers attached toOR lights can be used to estimate the direction of light and theorientation and/or trajectory of shadows in the OR or a room. If theorientation and/or trajectory of shadows in the OR or the room is known,virtual shadowing or shading with the same or similar orientation ortrajectory can be applied to virtual data display by the OHMD.

For example, one or more optical markers including one or more geometricshapes, geometric patterns, alphabetic, numeric, alphanumeric, and othercodes or patterns including bar codes and QR codes or combinationsthereof can be attached to a medial femoral epicondyle, for exampleusing a pin or a screw or an adhesive. An image and/or video capturesystem integrated into, attached to or separate from the OHMD can beused to monitor the position, and/or orientation and/or alignment and/ordirection of movement of the optical marker in relationship to the imageand/or video capture system; as the distal femur moves, the image and/orvideo capture system can detect the marker, for example based on itspre-programmed geometric shape, geometric pattern, alphabetic, numeric,alphanumeric, and other codes or patterns including bar codes and QRcodes, and can monitor and, optionally, record the movement. If a secondoptical marker, including one or more geometric shapes, geometricpatterns, alphabetic, numeric, alphanumeric, and other codes or patternsincluding bar codes and QR codes or combinations thereof is attached tothe lateral femoral condyle in the same example, the image and/or videocapture system can also monitor and, optionally record the position,and/or orientation and/or alignment and/or direction of movement of thesecond optical marker in relationship to the image and/or video capturesystem; by monitoring the position, and/or orientation and/or alignmentand/or direction of movement of the first optical marker on the medialfemoral epicondyle and the position, and/or orientation and/or alignmentand/or direction of movement of the second optical marker on the lateralfemoral epicondyle, the image and/or video capture system and relatedimage processing and pattern recognition software can also monitor and,optionally, record the movement of the femoral epicondylar axis, forexample during flexion and extension of the knee. One or more opticalmarkers including one or more geometric shapes, geometric patterns,alphabetic, numeric, alphanumeric, and other codes or patterns includingbar codes and QR codes or combinations thereof can be attached to aproximal tibia, e.g. an anterior tibial rim, a medial and/or lateraltibial spine, a lowest point of a medial plateau and/or a highest pointof a lateral tibial plateau, for example in the same example. The imageand/or video capture system integrated into, attached to or separatefrom the OHMD can be used to monitor the position, and/or orientationand/or alignment and/or direction of movement of the optical marker(s)attached to the tibia in relationship to the image and/or video capturesystem and in relationship to one or more femoral optical markers,thereby monitoring and, optionally recording, tibiofemoral motion, e.g.during a surgery. One or more optical markers including one or moregeometric shapes, geometric patterns, alphabetic, numeric, alphanumeric,and other codes or patterns including bar codes and QR codes orcombinations thereof can be attached to a patella, e.g. a most superioraspect, a most inferior aspect, a most lateral aspect and/or a mostmedial aspect, for example in the same example. The image and/or videocapture system integrated into, attached to or separate from the OHMDcan be used to monitor the position, and/or orientation and/or alignmentand/or direction of movement of the optical marker(s) attached to thepatella in relationship to the image and/or video capture system and inrelationship to one or more femoral optical markers, thereby monitoringand, optionally recording, patellofemoral motion, e.g. during a surgery.The image and/or video capture system integrated into, attached to orseparate from the OHMD can be used to monitor the position, and/ororientation and/or alignment and/or direction of movement of the opticalmarker(s) attached to the patella in relationship to the one or moretibial optical markers, thereby monitoring and, optionally recording,patellar motion in relationship to the tibia, e.g. during tibialadduction or abduction.

In some embodiments of the invention, an optical marker, for examplewith one or more specific geometric shapes, geometric patterns,alphabetic, numeric, alphanumeric, and other codes or patterns includingbar codes and QR codes or combinations thereof, can be assigned to avirtual surgical step. The marker can, for example, include written textdefining the surgical step or corresponding to the surgical step, whichcan be the immediately preceding surgical step or the next surgicalstep, for example in a virtual surgical plan. In some embodiments, thetext can be a number, for example a number corresponding to a particularsurgical step, e.g. 1—for distal femoral cut, 2—for anterior femoralcut, 3—for posterior femoral cut, 4—for first chamfer cut, 5—for secondchamfer cut. The number can be recognized by the image and/or videocapture system, which can then display the virtual view for thecorresponding surgical step, e.g. for 1—a cut plane for the distalfemoral cut or a virtual outline of the corresponding physical distalfemoral cut block. A combination of numbers and text can be used and theimage and/or video capture system and associated software and optionalpattern recognition software and systems can recognize the numbers andtext and trigger a command to display the corresponding virtual view ofthe corresponding virtual surgical step, e.g. 1F—distal femoral cut,2F—anterior femoral cut, 1T—proximal tibial cut, 2T—tibial keel punchetc.

In another example, an optical marker with one or more specificgeometric shapes, geometric patterns, alphabetic, numeric, alphanumeric,and other codes or patterns including bar codes and QR codes orcombinations thereof can be assigned to the step “distal femoral cut” ina virtual surgical plan for a total knee replacement in a patient; theoptical marker can include the text “distal femoral cut”. The surgeoncan, for example, affix the marker to the cut bone surface of the distalfemur or somewhere adjacent to it. An image and/or video capture systemintegrated into, attached to or separate from an OHMD can detect theoptical marker with the one or more specific geometric patterns and/orspecific geometric shapes assigned to “distal femoral cut”, indicatingthat the distal femoral cut has been completed; the image capture signalcan then initiate a command to the OHMD to display the next surgicalstep, e.g. an anterior cut plane or an outline of an anterior cut blockor cut guide, as the surgeon prepares to perform the next cut, e.g. theanterior femoral cut in this example.

In some embodiments, an optical marker, for example with one or morespecific geometric shapes, geometric patterns, alphabetic, numeric,alphanumeric, and other codes or patterns including bar codes and QRcodes or combinations thereof, can be integrated into, included in, orattached to a surgical instrument used for a surgical step in a virtualsurgical plan. For example, the optical marker can be included in,integrated into or attached to a surgical cut block or cutting tool,e.g. for a proximal tibial cut. Optionally, the marker can includewritten text defining the surgical step or corresponding to the surgicalstep, e.g. in a virtual surgical plan. In the immediately foregoingexample, an optical marker with one or more specific geometric shapes,geometric patterns, alphabetic, numeric, alphanumeric, and other codesor patterns including bar codes and QR codes or combinations thereof canbe assigned to the step “proximal tibial cut” in a virtual surgical planfor a total knee replacement in a patient; the optical marker caninclude the text “proximal tibial cut” which the surgeon can read andensure that the correct marker is used for the next surgical step thathe or she is contemplating, in this example a proximal tibial cut.

As the optical marker enters the surgeon's field of view, an imageand/or video capture system integrated into or attached to the OHMD onthe surgeon's head can detect the optical marker and display the nextvirtual surgical step, e.g. an outline of a virtual proximal tibial cutblock corresponding to the physical proximal tibial cut block, so thatthe surgeon can align or superimpose the physical surgical cut block orinstrument onto the outline of the virtual surgical cut block orinstrument. Alternatively, as the optical marker enters the surgeon'sfield of view, an image and/or video capture system integrated into orattached to the OHMD on the surgeon's head can detect the optical markerand display the next virtual surgical step, e.g. a virtual cut planewith a predetermined resection level, varus or valgus angle and/orslope, so that the surgeon can align or superimpose the physicalsurgical cut block and/or the physical surgical saw with the virtual cutplane. Once the surgical step is completed, e.g. a proximal tibial cut,and the surgeon removes the physical surgical instrument with theintegrated, included or attached optical markers from the surgical fieldand/or the field of view of the image and/or video capture system, theimage and/or video capture system can detect that the optical marker isnot present in the field of view anymore and software can generate acommand to turn off the display of OHMD or the display of the completedvirtual surgical step. Optionally, a command can be generated at thistime, optionally automatically, to display the next surgical step, e.g.a tibial keel punch including, for example, setting tibial rotation.Alternatively, the display of the OHMD unit can display the nextsurgical step as the next surgical instrument with the correspondingoptical marker for the next surgical step enters the field of view, e.g.in the surgeon's hand.

In a similar example, an optical marker can be attached to an acetabularreamer used for hip replacement. An image and/or video capture systemintegrated into or attached to an OHMD can detect the optical marker asit enters the surgeon's field of view triggering a command to displaythe reaming axis or a virtual display of the reamer with the intendedalignment and/or direction for the reaming step; as the optical markerwith the surgical instruments exits the surgeon's field of view, theimage and/or video capture system can detect it triggering a command tostop the display of the reaming axis or virtual display of the reamer,optionally switching to the next surgical step.

In some embodiments, one or more optical markers can be included in,integrated into or attached to an insert for a cutting block or guide.The insert can be configured to fit into one or more slots or guideswithin the cutting block or guide for guiding a saw blade.Representative cutting blocks or guides are, for example, cutting blocksor guides used in knee replacement, shoulder replacement, hipreplacement, and ankle replacement. These cutting blocks or guides are,for example, used to remove bone at the articular surface to fit thepatient's bone to the bone facing side of an implant or implantcomponent. The insert can be designed to partially or substantially fillthe entire slot or guide, e.g. in x and y direction or x and z directionor y and z direction depending on the shape and/or design of the cuttingblock or guide. If the insert partially fills or substantially fills theslot or guide in x and y direction, the insert can be configured toextend beyond the slot or guide in z direction. If the insert partiallyfills or substantially fills the slot or guide in x and z direction, theinsert can be configured to extend beyond the slot or guide in ydirection. If the insert partially fills or substantially fills the slotor guide in y and z direction, the insert can be configured to extendbeyond the slot or guide in x direction. Any direction is possibleincluding oblique directions, orthogonal directions and non-orthogonaldirections depending on the configuration of the cutting block or guideand the associated slots or guides. Oblique slots can, for example, beused for chamfer cuts in total knee replacement or oblique talar cuts intotal ankle replacement.

The portion(s) of the insert that extend beyond the slot or guide can,for example, include one or more integrated or attached optical markers.If more than one optical marker are used, the optical markers can bearranged at predefined angles and locations, e.g. 90 degrees or lessthan 90 degrees or more than 90 degrees. The insert can have similardimensions to a representative saw blade used with the cutting block orguide. The insert can indicate the position, location, orientation,alignment and direction of travel for a saw blade that will subsequentlybe inserted. The surgeon can place the insert inside the slot or guideof the physical cutting block or guide and align the insert, forexample, with a virtual cut plane or a virtual outline of the insert orcutting block or guide projected by the OHMD onto the surgical site,e.g. a distal femur in total knee replacement or a proximal femur intotal hip replacement. Once the insert is substantially aligned and/orsuperimposed with the virtual cut plane, the virtual outline of theinsert or cutting block or guide, the surgeon can pin the physicalcutting block or guide onto the bone, thereby affixing the cutting blockor guide to the bone in a position where the virtual surgical plan, e.g.the virtual cut plane or virtual outline of the insert or cutting blockor guide is substantially aligned with the physical cut plane and or thephysical insert or cutting block or guide. The surgeon can then insertthe physical saw blade and perform the physical cut. The insert can beconfigured to have a shape substantially similar to the physical sawblade, serving as a dummy saw blade.

Alternatively, the surgeon can place the physical saw blade inside theslot or guide of the physical cutting block or guide and the surgeon canalign the physical saw blade, for example, with a virtual cut plane or avirtual outline of the saw blade or cutting block or guide projected bythe OHMD onto the surgical site, e.g. a distal femur in total kneereplacement or a proximal femur in total hip replacement. Once thephysical saw blade is substantially aligned and/or superimposed with thevirtual cut plane, the virtual outline of the saw blade or cutting blockor guide, the surgeon can pin the physical cutting block or guide ontothe bone, thereby affixing the cutting block or guide to the bone in aposition where the virtual surgical plan, e.g. the virtual cut plane orvirtual outline of the saw blade or cutting block or guide issubstantially aligned with the physical cut plane and or the physicalsaw blade or cutting block or guide. The surgeon can then advance thephysical saw blade and perform the physical cut.

Optical markers can be included in, integrated into or attached to thecutting block or guide or the insert, e.g. a dummy saw blade. Opticalmarkers can also be attached or affixed the saw blade. The opticalmarkers can include a text or alphanumeric code for the surgeon thatdesignates, for example, a specific surgical step, e.g. 1F—distalfemoral cut, 2F—anterior femoral cut, 1T—proximal tibial cut, 2T—tibialkeel punch etc. The optical markers can also include one or morespecific geometric shapes, geometric patterns, alphabetic, numeric,alphanumeric, and other codes or patterns including bar codes and QRcodes or combinations thereof. The one or more specific geometricshapes, geometric patterns, alphabetic, numeric, alphanumeric, and othercodes or patterns including bar codes and QR codes or combinationsthereof can be specific for the surgical step, corresponding, forexample, to the lettering or alphanumeric code that indicates thesurgical step to the surgeon. An image and/or video capture systemintegrated into, attached to or separate from the OHMD can detect theone or more specific geometric shapes, geometric patterns, alphabetic,numeric, alphanumeric, and other codes or patterns including bar codesand QR codes or combinations thereof as the optical marker(s) enters thefield of view; the specific geometric shapes, geometric patterns,alphabetic, numeric, alphanumeric, and other codes or patterns can berecognized using image processing and/or pattern recognition softwaretriggering, for example, a command to display corresponding virtualsurgical step in the OHMD superimposed onto the surgical field with theview angle for the surgeon aligned with the surgical field or targetanatomy or bone cut. When the cutting block or guide, the insert, e.g. adummy saw blade, or the physical saw blade with the optical marker isremoved, the image and/or video capture system can detect that theoptical marker is not present in the field of view any longer,triggering, for example a command to turn off the OHMD display or thedisplay of the completed surgical step or to switch to the display ofthe next surgical step and corresponding virtual display.

In some embodiments of the invention, one or more optical markers, e.g.at select angles, e.g. 90 degrees or less or more or parallel or on oneaxis, can be included in, integrated into or attached to a cutting blockor guide.

In some embodiments, one or more optical markers can be used inconjunction with a spinal surgery, e.g. a vertebroplasty, a kyphoplasty,a posterior spinal fusion, an anterior spinal fusion, a lateral spinalfusion and/or a disk replacement. For example one or more opticalmarkers can be included in, integrated into, or attached to a needle, apin, an awl, a feeler probe, a ball handle probe, a straight probe, acurved probe, a tap, a ratchet, a screw driver, a rod template, a rodinserter, a rod gripper, a bender, a plug starter, a compressor, adistractor, a break off driver, an obturator, a counter torque, a quickconnector, a driver, a retractor, a retracting frame, an implantpositioner, a caliper, a plate holder, a plate bender, a forceps and thelike. The foregoing list is only exemplary and not to be construedlimiting of the invention. The one or more optical markers can be usedto designate the patient's left side and the patient's right side and/orthey can be used to designate the patient's spinal level, using, forexample, one or more geometric shapes, geometric patterns, alphabetic,numeric, alphanumeric, and other codes or patterns that can be detectedwith an image and/or video capture system integrated into, attached toor separate from the OHMD and that can be recognized using imageprocessing and/or pattern recognition.

One or more optical markers can be used to determine the position,location, orientation, alignment and/or direction of a needle, a pin, anawl, a feeler probe, a ball handle probe, a straight probe, a curvedprobe, a tap, a ratchet, a screw driver, a rod template, a rod inserter,a rod gripper, a bender, a plug starter, a compressor, a distractor, abreak off driver, an obturator, a counter torque, a quick connector, adriver, a retractor, a retracting frame, an implant positioner, acaliper, a plate holder, a plate bender, a forceps, a mill, a saw, areamer, a broach, an impactor, a cutting or drilling block, and/or othersurgical instrument and/or trial implant and/or implant component withuse of an image and/or video capture system integrated into, attached toor separate from the OHMD. For example, after the initial registrationor any subsequent registration of the patient, the surgical site, theOHMD, optionally an image and/or video capture system integrated into,attached to or separate from the OHMD, the virtual data and/or the livedata of the patient have been performed, the image and/or video capturesystem can detect an optical marker included in, integrated into, and/orattached to the surgical instrument. Since the location, position,alignment and/or orientation of the optical marker on the surgicalinstrument are known and the dimensions, e.g. at least one of them, orgeometry of the surgical instrument are known, the image and/or videocapture system can track the optical marker and the surgical instrumentwith regard to its location, position, orientation, alignment and/ordirection of movement.

In another example, two or more optical markers can be integrated intoor attached to different, optionally defined locations along the longaxis of a needle, a pin, an awl, a feeler probe, a ball handle probe, astraight probe, a curved probe, a tap, a ratchet, a screw driver, a rodtemplate, a rod inserter, a rod gripper, a bender, a plug starter, acompressor, a distractor, a break off driver, an obturator, a countertorque, a quick connector, a driver, a retractor, a retracting frame, animplant positioner, a caliper, a plate holder, a plate bender, aforceps, a mill, a saw, a reamer, a broach, an impactor, a cutting ordrilling block, and/or other surgical instrument and/or trial implantand/or implant component, for example instruments or trial implants orimplant components in knee replacement or hip replacement. An imageand/or video capture system can detect the two or more optical markersand their respective location can be determined. With the location ofthe two or more optical markers captured and defined by the image and/orvideo capture system, the long axis of the needle, pin, awl, probe, tap,mill, saw, reamer, broach, impactor, and/or other surgical instrumentand/or trial implant and/or implant component can be determined; otheraxes can be determined in addition to the long axis or instead of thelong axis. With the location of the optical markers on the needle, pin,awl, probe, tap, mill, saw, reamer, broach, impactor, and/or othersurgical instrument and/or trial implant and/or implant component known,the long axis or other axis of the needle, pin, awl, probe, tap, mill,saw, reamer, broach, impactor, and/or other surgical instrument and/ortrial implant and/or implant component known and the dimensions of theneedle, pin, awl, probe, tap, mill, saw, reamer, broach, impactor,and/or other surgical instrument and/or trial implant and/or implantcomponent known, any portions of the needle, pin, awl, probe, tap, mill,saw, reamer, broach, impactor, and/or other surgical instrument and/ortrial implant and/or implant component hidden by the tissue, e.g. belowthe skin and/or inside or within muscle, can be estimated and canoptionally be displayed by the OHMD in addition to the virtual orintended path or projected path or any other aspects of a virtualsurgical plan. Rather than using two or more optical markers in theforegoing embodiment, an optical marker long enough or wide enough ordeep enough to define one or more axes of a needle, pin, awl, probe,tap, mill, saw, reamer, broach, impactor, and/or other surgicalinstrument and/or trial implant and/or implant component can also beused.

Optionally, when two or more optical markers are used included in,integrated into or attached to a surgical instrument, the opticalmarkers, can be arranged at the same angles, e.g. parallel or on thesame axis, or at different angles, e.g. orthogonal angles ornon-orthogonal angles. This can be particularly useful, when the opticalmarkers include one or more of a geometric shape, geometric pattern,alphabetic, numeric, alphanumeric, and other codes or patterns includingbar codes and QR codes or combinations thereof. By arranging the opticalmarkers and any associated geometric shapes, geometric patterns,alphabetic, numeric, alphanumeric, and other codes or patterns includingbar codes and QR codes or combinations thereof in this manner, theangular orientation of the surgical instrument can be determined in amore accurate manner. For example, at certain view angles from an imageand/or video capture system integrated into or attached to an OHMDselect geometric shapes, geometric patterns, alphabetic, numeric,alphanumeric, and other codes or patterns including bar codes and QRcodes or combinations thereof of a first optical marker on a surgicalinstrument may be only partially visualized or not visualized at all dueto the angular orientation; when a second optical marker is oriented ata different angle, location and/or orientation on the same surgicalinstrument, the view angle from the image and/or video capture systemintegrated into or attached to the OHMD to the second optical marker canallow for a complete or a more complete visualization of the one or moregeometric shapes, geometric patterns, alphabetic, numeric, alphanumeric,and other codes or patterns including bar codes and QR codes orcombinations thereof, thereby allowing a more accurate determination ofthe angular orientation of the second optical marker and, with that, thesurgical instrument. In addition, the respective projections of thefirst optical marker and/or the second optical marker measured by theimage and/or video capture system, optionally paired with any parallaxinformation when two or more cameras are used, e.g. one positioned nearthe left eye and another positioned near the right eye, can be used tomore accurately determine their relative position and the position ofthe surgical instrument.

An image and/or video capture system integrated into or attached to orseparate from an OHMD can detect an optical marker included in,integrated into or attached to a needle, a pin, an awl, a feeler probe,a ball handle probe, a straight probe, a curved probe, a tap, a ratchet,a screw driver, a rod template, a rod inserter, a rod gripper, a bender,a plug starter, a compressor, a distractor, a break off driver, anobturator, a counter torque, a quick connector, a driver, a retractor, aretracting frame, an implant positioner, a caliper, a plate holder, aplate bender, a forceps, a mill, a saw, a reamer, a broach an impactor,a cutting or drilling block, and/or other surgical instrument and/ortrial implant and/or implant component as it enters the surgeon's fieldof view triggering a command to display the predetermined path or planeor a virtual display of the a needle, a pin, an awl, a feeler probe, aball handle probe, a straight probe, a curved probe, a tap, a ratchet, ascrew driver, a rod template, a rod inserter, a rod gripper, a bender, aplug starter, a compressor, a distractor, a break off driver, anobturator, a counter torque, a quick connector, a driver, a retractor, aretracting frame, an implant positioner, a caliper, a plate holder, aplate bender, a forceps, a mill, a saw, a reamer, a broach, an impactor,a cutting or drilling block, and/or other surgical instrument and/ortrial implant and/or implant component or other display mode or type ofthe virtual surgical plan, for example with the intended position,location and/or alignment and/or direction for the intended surgicalstep; as the optical marker with the surgical instrument exits thesurgeon's field of view, the image and/or video capture system candetect it triggering a command to stop the display of the predeterminedpath or the virtual display of the surgical instrument or other aspectsof the virtual surgical plan, optionally switching to the next surgicalstep and corresponding virtual display. In a spinal procedure as well asselect other procedures, the next surgical step can involve the sameside of the patient or the opposite side of the patient at the samespinal level, where the corresponding virtual display for the nextsurgical step for a given level and side can be initiated by the OHMDdisplay. The next surgical step can involve the same side of the patientor the opposite side of the patient at an adjoining or different spinallevel, where the corresponding virtual display for the next surgicalstep for a given level and side can be initiated by the OHMD display.

Optical markers can include one or more QR codes. QR codes can be partof or can be embedded in a geometric pattern or geometric shape includedin an optical marker. Optical markers can be a QR code.

If an optical marker is attached to a surgical instrument, theattachment can occur in a defined location and/or position and/oralignment, for example at an end of the surgical instrument. Theattachment can include, for example, an opening with a stop therebydefining the location and/or position and/or alignment of the opticalmarker on the surgical instrument. For example, the optical marker canhave an opening with a stop that is large enough to accommodate thesurgeon facing end of a pin or drill, for example inserted into aspinous process or a facet joint or a portion of a pedicle. With thistype of attachment and other attachments that secure the marker in adefined location, position and/or orientation on the surgicalinstrument, an image and/or video capture system can detect the opticalmarker and its location, position and/or orientation can be used todetermine the location, position, and/or orientation of the surgicalinstrument, e.g. a pin, including its tip or frontal portion inside thepatient due to their defined spatial relationship and due to the knowngeometry of the surgical instrument.

In some embodiments of the invention, an optical marker can be used todetermine or identify the position, location, orientation, alignment,dimensions, axis or axes, plane or planes of a surgical alteration. Forexample, if a bone cut has been performed in a surgical step, one ormore optical markers can be attached to the cut bone to determine one ormore of its position, location, orientation, alignment, dimensions,shape, geometry, axis or axes, plane or planes. For example, one, two ormore optical markers can be placed near or attached to the periphery orthe edge of the cut bone or surgical alteration; an image and/or videocapture system integrated into, attached to or separate from the OHMDcan detect the location, position, and/or orientation of the opticalmarkers and software can be used, for example, to analyze the location,position, and/or orientation information of the optical markers toderive information on the periphery and/or edge and/or shape of the cutbone or surgical alteration. One, two or more optical markers can beplaced near or attached to the cut bone or surgical alteration; an imageand/or video capture system integrated into, attached to or separatefrom the OHMD can detect the location, position, and/or orientation ofthe optical markers and software can be used, for example, to analyzethe location, position, and/or orientation information of the opticalmarkers to derive information on the shape or geometry of the cut boneor surgical alteration. If the bone cut is planar, one or more opticalmarkers with a planar bone facing surface or one or more optical markersattached to a carrier or instrument, e.g. a plastic piece, with a planarbone facing surface can be held against, affixed to or attached to thecut bone surface; an image and/or video capture system integrated into,attached to or separate from an OHMD can then be used to detect the oneor more optical markers and software can be used, for example, toanalyze the location, position and/or orientation information of the oneor more optical markers to derive information on the location and/orposition and/or orientation and/or alignment of the plane of the bonecut, including for example in relationship to other anatomic landmarksand/or other optical markers. The carrier or instrument for the opticalmarker can be transparent or semi-transparent so that the surgeon cancheck or confirm that the carrier or instrument and the attached opticalmarker(s) are flush against the bone cut prior to determining orconfirming, for example, the plane of the bone cut. Once the plane ofthe bone cut has been determined or confirmed in this manner, theoptical marker(s) attached to the cut bone and/or the determined planeof the bone cut can be used to plan the next surgical alteration, e.g.the next bone cut or surgical alteration, e.g. an anterior or posteriorfemoral cut after the distal femoral cut in knee replacement, or achamfer cut after the anterior and posterior femoral cuts in kneereplacement, or a cut on an opposing articular surface. By determining,confirming and/or referencing a preceding surgical alteration, e.g. abone cut, in this manner, the accuracy of subsequent surgical steps canbe improved thereby ultimately improving the overall accuracy of thesurgical procedure.

In some embodiments of the invention, one or more optical markers can beattached to or affixed to a patient's thigh or distal femur. The one ormore optical markers can, for example, be attached to the skin of thedistal thigh, e.g. above the knee joint space. The attachment can beperformed using, for example, an adhesive that attaches the one or moreoptical markers to the patient's skin. The one or more optical markerscan optionally be sterile. The one or more optical markers canoptionally be magnetic. In this example, a magnetic base can optionallybe attached to the patient's skin, for example using an adhesive. Asurgical drape which can be transparent, semi-transparent or nottransparent can then be placed over the magnetic base and the magneticoptical marker can then be attached to the magnetic base attached to thepatient's skin. Alternatively, once a skin incision is made, one or moreoptical markers can be rigidly attached to one or more bones, e.g. thedistal femur and/or the proximal tibia. The rigid attachment can be doneusing pins or screws or other attachment mechanisms.

An image and/or video capture system integrated into, attached to orseparate from the OHMD can register the location and/or position and/ororientation and/or alignment of the one or more optical markers, forexample while the leg is in neutral position and/or extension and/or anyother position, including arbitrary positions or positions chosen by thesurgeon and/or operator. The surgeon and/or operator can then move theleg and thigh into multiple different positions and/or orientationsand/or alignments and/or the surgeon and/or operator can move the legand thigh in circular fashion or semicircular fashion. An image and/orvideo capture system integrated into, attached to or separate from theOHMD can register the location and/or position and/or orientation and/oralignment of the one or more optical markers for these multipledifferent positions and/or orientations and/or alignments of the leg orthigh and/or during the different circular or semicircular movements.The resultant information can be used to determine the center ofrotation, which, in this example, can be the center of the hip joint.

In some embodiments, an ankle clamp can be applied to the ankle of thepatient's leg. The ankle clamp can include one or more optical markersincluding, for example, one or more QR codes. The ankle clamp and/or theoptical markers can be disposable. The ankle clamp and the integrated orattached optical markers can be used to determine the position of themedial and lateral malleolus and with that, for example, the center or ⅓or ⅔ distance points between the malleoli of the ankle joint using animage and/or video capture system integrated into, attached to orseparate from the OHMD. Alternatively, one or more optical markers canbe applied to medial and/or lateral malleolus. In some embodiments ofthe invention, a magnetic base can be affixed to the medial and lateralmalleolus. The ankle can then be prepped and draped in sterile techniqueand one or more sterile, magnetic optical markers can be applied overthe drape or surgical cover affixing the one or more optical markers tothe magnetic base with the interposed drape or surgical cover. An imageand/or video capture system integrated into, attached to or separatefrom the OHMD can then be used to identify the optical markers over themedial and lateral malleolus and the center, ⅓ or ⅔ distance points ofthe ankle joint.

With the center of the hip joint determined using the one or moreoptical markers on the thigh or distal femur and the center or ⅓ or ⅔distance points of the ankle joint determined using the ankle clampand/or one or more optical markers, the system can derive the patient'smechanical axis and any surgical interventions, e.g. correction of varusor valgus deformity with corresponding femoral and/or tibial and/ortalar bone cuts can be planned and subsequently projected using theOHMD.

In some embodiments of the invention, one or more optical markers can beattached to or affixed to a patient's arm. The one or more opticalmarkers can, for example, be attached to the skin of the upper armthigh, e.g. above the elbow. The attachment can be performed using, forexample, an adhesive that attaches the one or more optical markers tothe patient's skin. The one or more optical markers can optionally besterile. The one or more optical markers can optionally be magnetic. Inthis example, a magnetic base can optionally be attached to thepatient's skin, for example using an adhesive. A surgical drape whichcan be transparent, semi-transparent or not transparent can then beplaced over the magnetic base and the magnetic optical marker can thenbe attached to the magnetic base attached to the patient's skin.Alternatively, once a skin incision is made, one or more optical markerscan be rigidly attached to one or more bones, e.g. the proximal humerus.The rigid attachment can be done using pins or screws or otherattachment mechanisms.

An image and/or video capture system integrated into, attached to orseparate from the OHMD can register the location and/or position and/ororientation and/or alignment of the one or more optical markers, forexample while the arm is in neutral position and/or extension and/orabduction and/or any other position, including arbitrary positions orpositions chosen by the surgeon and/or operator. The surgeon and/oroperator can then move the arm into multiple different positions and/ororientations and/or alignments and/or the surgeon and/or operator canmove the arm in circular fashion or semicircular fashion. An imageand/or video capture system integrated into, attached to or separatefrom the OHMD can register the location and/or position and/ororientation and/or alignment of the one or more optical markers forthese multiple different positions and/or orientations and/or alignmentsof the arm and/or during the different circular or semicircularmovements. The resultant information can be used to determine the centerof rotation, which, in this example, can be the center of rotation ofthe shoulder joint.

In some embodiments of the invention, one or more optical markers can beattached to an operating room (OR) table. If the optical marker isparallel to the OR table, a single marker can be sufficient to determinethe principal plane of the OR table, e.g. the horizontal plane, whichcan be the plane on which the patient is resting, for example in supine,prone, lateral or oblique or other positions known in the art. This canbe aided by using optical markers that include a surface or plane thatis parallel or perpendicular or at a defined angle to the OR table andthat is large enough to be detected by the camera, image or videocapture system integrated into, attached to or separate from the OHMD.For example, such a plane of the optical marker can measure 1×1 cm, 2×2cm, 2×3 cm, 4×4 cm, 4×6 cm and so forth. Alternatively, multiple, e.g.two, three or more, optical markers can be used to determine a planethrough the markers corresponding to the principal plane of the OR tableor a plane parallel to the principal plane of the OR table or, forexample, a plane vertical to the OR table or, for example, a plane at adefined angle to the OR table. If the OR table is hidden by surgicaldrapes, one or more magnetic or otherwise attachable bases can beattached to the OR table prior to placing the drapes. After the drapeshave been placed, one or more magnetic or otherwise attachable opticalmarkers can be affixed to the magnetic bases or attachment mechanismswith the interposed surgical drapes. Alternatively, one or more holdingarms or extenders of known geometry can be attached to the OR table andone or more optical markers can be attached to or can be integrated intothe holding arms or extenders. An image and/or video capture systemintegrated into, attached to or separate from the OHMD can then identifythe location, position, orientation and/or alignment of the one or moreoptical markers. The resultant information can be used to determine theprincipal plane of the OR table on which the patient is lying. One ormore OHMDs can be referenced using, for example, an image and/or videocapture system integrated into or attached to the OHMD relative to theOR table and/or the attached optical markers. Once the principal planeof the OR table is determined in the system, virtual surgical steps canbe planned in the virtual surgical plan of the patient in relationshipto the principal plane of the OR table. For example, one or more bonecuts can be planned and/or performed perpendicular to the principalplane of the OR table, for example with the patient in supine or proneposition or any other desired position. One or more bone cuts can beplanned and/or performed at defined angles other than 90 degreesrelative to the horizontal plane of the OR table, for example with thepatient in supine or prone position or any other desired position. Oneor more bone cuts can be planned and/or performed at a non-orthogonalplane or orientation relative to the principal plane or horizontal planeof the OR table, for example with the patient in supine or proneposition or any other desired position, optionally referencing a planevertical to the OR table, displayed by the OHMD. The principal plane ofthe OR table can be used as a reference in this manner including forcomparing or referencing virtual data of the patient and live data ofthe patient and including for comparing or referencing a virtualsurgical plan. Such bone cuts at orthogonal angles or non-orthogonalangles, e.g. relative to the OR table or relative to anatomy, anatomiclandmarks, anatomic or biomechanical axes of the patient, can beexecuted using one or more virtual surgical guides or cut blocks and/orone or more physical surgical guides or cut blocks. Virtual surgicalguides or cut blocks can include one or more dimensions corresponding tophysical surgical guides or cut blocks.

One or more optical markers attached to or referencing the OR table canalso serve as a fixed reference for the one or more OHMDs during asurgical procedure. This can be useful, for example, when the patientand/or the extremity and/or the surgical site moves during theprocedure. A fixed reference to the OR table can aid in maintainingregistration of the one or more OHMDs and the virtual surgical plan andthe live data of the patient and/or OR.

In some embodiments of the invention, one or more optical markers can beplaced on or attached to the patient in the area of the surgical fieldand/or in an area away from the surgical field. An image and/or videocapture system integrated into, attached to or separate from the OHMDcan be used to identify the one or more optical markers and to determinetheir location, position, orientation and/or alignment. The image and/orvideo capture system can also, optionally, determine the location,position, orientation and/or alignment of one or more optical markersattached to or referencing the OR table. The system can reference thecoordinates and/or the spatial relationship of the one or more opticalmarkers attached to the patient in the area of the surgical field and/orin an area away from the surgical field and the one or more opticalmarkers attached to or referencing the OR table. In this manner, if thepatient's body moves during the procedure, e.g. during a broaching of aproximal femur or an acetabular reaming during hip replacement, or afemoral or tibial component impacting during knee replacement, or duringa pinning or cutting of a bone, or during a placement of a spinaldevice, e.g. a cage or a pedicle screw, the movement between the one ormore optical markers attached to the patient in the area of the surgicalfield and/or in an area away from the surgical field and the one or moreoptical markers attached to or referencing the OR table and the changein coordinates of the one or more optical markers attached to thepatient in the area of the surgical field and/or in an area away fromthe surgical field can be detected and the amount of movement, directionof movement and magnitude of movement can be determined; the resultantinformation can, for example, be used to update or adjust or modify avirtual surgical plan or to update or adjust or modify the display ofthe virtual surgical plan or virtual surgical steps or virtual displaysfor the movement of the patient, including for example by updating,moving or adjusting one or more aspects or components of the virtualsurgical plan including one or more of a virtual surgical tool, virtualsurgical instrument including a virtual surgical guide or cut block,virtual trial implant, virtual implant component, virtual implant orvirtual device, a predetermined start point, predetermined startposition, predetermined start orientation or alignment, predeterminedintermediate point(s), predetermined intermediate position(s),predetermined intermediate orientation or alignment, predetermined endpoint, predetermined end position, predetermined end orientation oralignment, predetermined path, predetermined plane, predetermined cutplane, predetermined contour or outline or cross-section or surfacefeatures or shape or projection, predetermined depth marker or depthgauge, predetermined angle or orientation or rotation marker,predetermined axis, e.g. rotation axis, flexion axis, extension axis,predetermined axis of the virtual surgical tool, virtual surgicalinstrument including virtual surgical guide or cut block, virtual trialimplant, virtual implant component, implant or device, non-visualizedportions for one or more devices or implants or implant components orsurgical instruments or surgical tools, and/or one or more of apredetermined tissue change or alteration using the new patientcoordinates or the new coordinates of the surgical field.

In some embodiments of the invention, portions of the optical marker orthe entire optical marker can be radiopaque, so that the optical markercan also be visible on a radiograph or other imaging studies thatutilize ionizing radiation including, for example, fluoroscopy, digitaltomosynthesis, cone beam CT, and/or computed tomography. Differentlevels or degrees of radiopacity can be present in different portions orareas of the optical marker. Different levels or degrees of radiopacitycan be utilized to encode information. For example, different levels ofradiopacity can be used to encode information also contained, forexample, in an optically readable alphanumeric code, bar code or QR. Thedifferent levels of radiopacity can optionally be arranged in a bar likethickness distribution, which can optionally mirror portions or all ofthe information contained in a bar code. The different levels ofradiopacity can optionally be arranged in a point or square likethickness distribution, which can optionally mirror portions of theinformation contained in a QR code. Different radiopacity can beobtained by varying the thickness of the metal, e.g. lead. Radiopaqueoptical markers with information encoded in such manner can, forexample, be manufactured using 3D metal printers. They can also be CNCmachined, e.g. from bar stock or cast blanks.

The radiopaque portions of the optical marker can include information onlaterality, e.g. L for left and R for right, visible on the radiograph,for example through different material thicknesses, e.g. lead; the sameinformation can be included in an attached alphanumeric code or text,bar code or QR code which can be read by a bar code or QR code reader oran image and/or video capture system integrated into, attached to orseparate from the OHMD. The radiopaque portions of the optical markercan include information on anatomical site, e.g. L5 or L4, T1 or T2, C3or C7, knee, hip, visible on the radiograph, for example throughdifferent material thicknesses, e.g. lead; the same information can beincluded in an attached alphanumeric code or text, bar code or QR codewhich can be read by a bar code or QR code reader or an image and/orvideo capture system integrated into, attached to or separate from theOHMD. Image processing techniques and/or software can be applied to theradiographic information including the optical marker andradiographically encoded information such as laterality and/or site andthe information included in the radiograph can be compared against theinformation included on the optical scan. If any discrepancies aredetected, an alert can be triggered, which can, for example, bedisplayed in the OHMD.

Multiple partially or completely radiopaque optical markers can be used.The radiopaque optical markers can be applied at different locations andin different planes around the surgical site. In spinal surgery, forexample, one, two, three or more radiopaque optical markers can beapplied to the skin around the spinal levels for the intended surgery;one, two, three or more radiopaque optical markers can be attached to apin, drill or screw inserted into a spinous process and/or a pedicle orother spinal element; one, two, three or more radiopaque optical markerscan be applied to the patient's flank or abdomen. In hip replacementsurgery, one, two, three or more radiopaque optical markers can beapplied to the anterior superior iliac spine on the patient's intendedsurgical side, e.g. with an adhesive to the skin or attached to a pin ordrill to the bone; one, two, three or more radiopaque optical markerscan be applied to the anterior superior iliac spine on the patient'scontralateral side, e.g. with an adhesive to the skin or attached to apin or drill to the bone; one, two, three or more radiopaque opticalmarkers can be applied to the symphysis pubis, e.g. with an adhesive tothe skin or attached to a pin or drill to the bone; one, two, three ormore radiopaque optical markers can be applied to the acetabulum on thepatient's intended surgical side, e.g. attached to a pin or drill to thebone; one, two, three or more radiopaque optical markers can be appliedto the greater trochanter on the patient's intended surgical side, e.g.attached to a pin or drill to the bone. By using multiple radiopaqueoptical markers in multiple different locations and in different planesaround the surgical site, the accuracy of any three-dimensional spatialregistration and cross-reference of the optical markers in differentmodalities, e.g. radiographs, image capture, can be increased, forexample by obtaining multiple x-rays at different angles, e.g. AP,lateral and/or oblique, and/or by imaging the radiopaque optical markersfrom multiple view angles using an image and/or video capture systemintegrated into, attached to or separate from the OHMD. By usingmultiple optical markers in multiple different locations and indifferent planes around the surgical site, the accuracy of anythree-dimensional spatial registration of the optical markers can beincreased, for example by imaging the optical markers from multiple viewangles using an image and/or video capture system integrated into,attached to or separate from the OHMD. In addition, the accuracy of theregistration can be better maintained as the view angle or radiographicangle changes, for example during the course of the surgical procedureor due to patient movement.

In some embodiments of the invention, the system performance can betested. System performance tests can, for example, measure a phantomincluding two or more optical markers at known locations, positions,orientations and/or alignment. With the coordinates of the two or moreoptical markers known along with the distance(s) and angle(s) betweenthe markers, the accuracy of performing distance measurements and/orangle measurements and/or area measurements and/or volume measurementsusing an image and/or video capture system integrated into, attached toor separate from the OHMD can be determined. In addition, by repeatingthe measurements, the reproducibility and/or precision of performingdistance measurements and/or angle measurements and/or area measurementsand/or volume measurements using an image and/or video capture systemintegrated into, attached to or separate from the OHMD can bedetermined. The accuracy and/or the reproducibility and/or the precisionof performing distance measurements and/or angle measurements and/orarea measurements and/or volume measurements using an image and/or videocapture system integrated into, attached to or separate from the OHMDcan be determined for static and dynamic conditions. Static conditionscan be conditions where a patient, a spine, an extremity, a joint and/ora bone do not move. Dynamic conditions can be conditions where apatient, a spine, an extremity, a joint and/or a bone move during theimage capture. Dynamic conditions can, for example, be useful indetermining the center of rotation of a joint. Measurements for staticconditions and for dynamic conditions can be performed for differentview angles and distances of the image and/or video capture systemintegrated into, attached to or separate from the OHMD. Measurements forstatic conditions and for dynamic conditions can be performed with theOHMD at rest, not moving. Measurements for static conditions and fordynamic conditions can be performed with the OHMD not at rest, butmoving, for example moving with the operators head.

TABLE 5 shows exemplary tests with various combinations of testconditions and test parameters for which the accuracy and thereproducibility and/or the precision of the measurements can bedetermined. Any combination is possible. Other parameters, e.g.reproducibility of color temperature (e.g. in Kelvin), can be measured.Other statistical tests can be applied. All measurements and allstatistical determinations and parameters can be assessed for static,dynamic, OHMD at rest and OHMD moving conditions including at differentangles and distances of the image and/or video capture system to thetarget anatomy and/or test apparatus and/or phantom.

Volume Coor- Area enclosed dinates Distance Angle enclosed Volume by ofbetween between by of multiple optical optical optical optical opticaloptical markers markers markers markers marker(s) markers Accuracy X X XX X X Reprodu- X X X X X X cibility/ Precision Static X X X X X XDynamic X X X X X X OHMD X X X X X X at rest OHMD X X X X X X moving

Once the accuracy and/or the reproducibility and/or the precision ofperforming distance measurements and/or angle measurements and/or areameasurements and/or volume measurements and/or coordinate measurementsusing an image and/or video capture system integrated into, attached toor separate from the OHMD has been determined, threshold values can, forexample, be defined that can indicate when the system is operatingoutside a clinically acceptable performance range. The threshold valuescan be determined using standard statistical methods known in the art.For example, when a view angle and/or a distance or a movement speed ofan image and/or video capture system integrated into an OHMD indicatethat a measurement value can fall outside two standard deviations of thesystem performance including overall system performance, it can triggeran alert to the surgeon that the display of virtual data, e.g. portionsof a virtual surgical plan, virtual projected paths or virtual planes,e.g. virtual cut planes, may not be accurate. A binary, e.g. yes, no,system can be used for triggering an alert that the image and/or videocapture system and/or the OHMD display are operating outside aclinically acceptable performance range, e.g. exceeding certain viewangles, exceeding or being below certain distances to the targetanatomy, or exceeding an acceptable movement speed. Alternatively, asliding scale can be used as the system enters progressively into arange outside the clinically acceptable performance range. The slidingscale can, for example, be a color scale from green to red with mixedcolors in between. The sliding scale can be an acoustic signal thatincreases in intensity or frequency the further the system operatesoutside the clinically acceptable range. The sliding scale can be avibration signal that increases in intensity or frequency the furtherthe system operates outside the clinically acceptable range. In someembodiments of the invention, the OHMD can optionally turn off thedisplay of any virtual data of the patient, e.g. virtual planinformation, virtual surgical guides or cut blocks or virtual planes orintended paths, when one or more test data indicate that the system isoperating outside its clinically acceptable performance range. When testdata indicate that the system is operating again inside the clinicallyacceptable performance range, the OHMD display can turn back on. Systemtests including accuracy tests and reproducibility tests can beperformed intermittently, e.g. every 3 seconds, 5 seconds, 10 seconds,20 seconds, 30 seconds, 1 minutes, 2 minutes and so forth. System testscan be performed continuously. System tests can be performedintermittently or continuously but limited to times when virtual dataare displayed by the OHMD. System tests can be performed intermittentlyor continuously but limited to times when surgical steps that requirehigh accuracy or reproducibility are being performed. Such stepsrequiring high accuracy or high reproducibility can be identified forexample by the surgeon through voice commands or other commands or theycan be identified in the virtual surgical plan, e.g. automatically or bysurgeon choice.

In some embodiments of the invention radiopaque and non-radiopaqueoptical markers can optionally be attached to or applied to extendersthat increase the distance of the optical marker from the patient'sskin. Such extenders can, for example, be anchored in a spinous process,a pedicle or other spinal element via a pin, drill or screw. The use ofextenders with attached radiographic optical markers can increase theaccuracy of registration between radiographic data and image capturedata, for example when AP and lateral radiographs are used. The use ofextenders with attached optical markers can help define anatomic orinstrument axes and other information when image capture is used. Whentwo or more markers are used with extenders and the markers areseparated by a distance greater than the spatial resolution of the imageand/or video capture system, the accuracy in determining, for example,an axis between the two markers can increase, for example as the lengthof the extender and the distance between the markers increases.

Optical markers can be visible with other imaging modalities, e.g. MRI,nuclear scintigraphy, SPECT or PET. Optical markers can, for example, bedoped with an MRI contrast agent such as Gadolinium-DTPA so that theyare MRI visible. Optical markers can, for example, be doped with anisotope or positron emitter so that they are SPECT or PET visible.

When an optical marker includes a QR code or when a QR code is used asan optical marker, it can also address inventory management issues andquality concerns before, during and after surgery. Operating the wrongside of a patient is a common quality problem related to surgery, whichcan have devastating consequences for the patient. Similarly, in spinalsurgery, operating the wrong spinal level can result in serious injuryof the patient. Optical markers used for determining the location,position, orientation, alignment and/or direction of travel, ifapplicable, of a patient, a limb, a joint, a surgical site, a surgicalinstrument, a trial implant and/or an implant component can also includeinformation any of the following using, for example, bar codes or QRcodes included in, integrated into or attached to the optical marker:

-   -   Patient identifiers    -   Patient demographics, e.g. age, sex, height, BMI    -   Patient medical history    -   Patient risk factors    -   Patient allergies    -   Side to be operated, e.g. left vs. right    -   Site to be operated, e.g. knee vs. hip, spinal level L1 vs. L2,        etc.    -   Spinal level(s) to be operated    -   Portions of virtual surgical plan, e.g.        -   resection amounts,        -   resection levels for a given surgical step,        -   position and/or orientation of bone cuts        -   slope of a tibial cut,        -   implant rotation, e.g. femoral component rotation, tibial            component rotation        -   implant flexion, e.g. femoral component flexion        -   intended depth, location, position, orientation, direction,            coordinates of burring        -   intended depth, location, position, orientation, direction,            coordinates of reaming        -   intended depth, location, position, orientation, direction,            coordinates of milling        -   angle of a femoral neck cut,        -   acetabular angle,        -   acetabular anteversion,        -   femoral anteversion,        -   offset        -   femoral shaft axis        -   intended implant component axes/alignment        -   intended polyethylene components, thickness (e.g. hip            acetabular liner, knee tibial inserts, shoulder glenoid            inserts)    -   Templating or sizing related information        -   Size of selected implant component, e.g. knee femoral,            tibial or patellar component, hip acetabular shell,            acetabular liner, femoral stem, femoral head, with mobile            bearing components femoral neck portion        -   Side of implant component, left vs. right    -   Inventory management information, e.g.        -   Version, type, model of instrument used        -   Lot number of instrument used        -   Place of manufacture of instrument used        -   Date of manufacture of instrument used        -   Date of first sterilization of instrument used        -   Number of sterilization cycles applied to instrument used        -   Date of last sterilization of instrument used        -   Date instrument delivered to hospital or surgery center        -   Version, type, model of implant component used        -   Lot number of implant component used        -   Place of manufacture of implant component used        -   Date of manufacture of implant component used        -   Date of sterilization of implant component used        -   Date implant component delivered to hospital or surgery            center        -   Any other information relevant to inventory management

Optionally, QR codes that include some of this information can also beseparate from the optical marker. In some embodiments of the invention,separate bar code and/or QR code readers can be used prior to, duringand/or after the surgery to read the information included on the barcodes and/or QR codes. In some embodiments of the invention, an imageand/or video capture system integrated into or attached to or separatefrom the OHMD can be used to read the information included on the barcodes and/or QR codes. The information read from the bar code and/or QRcode can then, for example, be compared against portions of the virtualsurgical plan and/or, for example, the physical patient's side preparedfor surgery, e.g. left vs. right, the physical patient site prepared forsurgery, e.g. spinal level L4 vs. L5 (as seen, for example, onradiographs), the physical surgery executed, the physical instrumentselected, the physical implant trial selected, the physical implantcomponent selected.

When a pin or a screw is placed in a surgical site including a jointand/or a bone, for example also in a spinal level, e.g. a spinousprocess or pedicle, with an integrated or attached optical marker with aQR code or when an instrument, a trial implant, and/or an implantcomponent with an integrated or attached optical marker with a QR codeenters the field of view of a bar code and/or QR code reader and/or animage and/or video capture system integrated or attached to the OHMD, orenters the proximity of the surgical field or surgically altered tissue,the information on the bar code or QR code on the physical pin or screw,the physical instrument, the physical trial implant, and/or the physicalimplant component can be read and compared against the intended surgicalsite information and/or the intended laterality information and/or thevirtual surgical plan and/or the intended sizing information and/or theintended templating information. In the example of a spinal level, thebar code and/or QR code reader and/or the image and/or video capturesystem integrated or attached to the OHMD, can read the QR codeidentifying the intended spinal level and side (left vs. right) for apin or a pedicle screw or other device(s). The information can becompared to the virtual surgical plan of the patient and/or x-rayinformation. For example, intra-operative x-rays can be used by thesystem to automatically or semi-automatically or user-operated identifyspinal levels, e.g. counting up from the sacrum, e.g. by detecting thesacral endplate and opposing endplates and/or pedicles. If the systemdetects a discrepancy in spinal level or laterality between theinformation read from the pin, screw or device and the integrated orattached optical marker and bar code or QR code, the virtual surgicalplan and/or the radiographic information, it can trigger an alert tocheck the device, check the surgical plan, and/or to re-confirm thespinal level and/or side. The foregoing example is not limited toradiographic information; other imaging tests known in the art, e.g. CT,MRI, etc., can be used for determining or identifying the anatomic siteand side, including for spinal levels.

If the reading of the QR code indicates a discrepancy in any of theinformation embedded in the QR code, e.g. site, laterality, level,portions or aspects of virtual surgical plan, sizing or templatinginformation, vs. the physical live data during the surgery, e.g. thephysical position or spinal level or laterality of the inserted pin orscrew, the physical instrument used, the physical trial implant used,and/or the physical implant component used, an alert can be triggered,for example in the OHMD or on a computer monitor used for planning,display, or modifying the virtual surgical plan. The alert can bevisual, e.g. red warning signs or stop signs or alert signs displayed,or acoustic, or a vibration, or combinations thereof. Any other alertknown in the art can be used.

For example, when a surgeon is operating on a patient to replace thepatient's left knee, one or more implant components or an attachedholder or packaging label or sterile package can include an opticalmarker including a QR marker. The QR marker can indicate the laterality,e.g. left femoral component vs. right femoral component. If the scrubtechnician accidentally hands the surgeon a right femoral component forimplantation into the patient's left knee, an image and/or video capturesystem integrated or attached to the OHMD that the surgeon is wearingcan read the QR code as the surgeon takes the femoral component and asthe femoral component with the attached optical marker and QR codeenters the surgeon's field of view or enters the proximity of thesurgical field. The image and/or video capture system and related systemsoftware can read the QR code identifying that the implant component isfor a right knee; the system software can then compare the informationto the virtual surgical plan of the patient or the templating and/orsizing information which can indicate that a left knee was planned, thentriggering an alert that an incorrect femoral component has entered thefield of view of the surgeon or has entered into the proximity of thesurgical field, as for example demarcated by another optical marker. Thealert can assist the surgeon in correcting the error by switching to thecorrect side component.

In another example, when a surgeon is operating on a patient to replacethe patient's left knee, one or more implant components or an attachedholder or packaging label or sterile package can include an opticalmarker including a QR marker. The QR marker can indicate the size of theimplant component, e.g. size 5 or 6 or other femoral component or size 5or 6 or other tibial component or size 2 or 3 or other patellarcomponent. If the scrub technician accidentally hands the surgeon a size4 femoral component for implantation into the patient's which has beentemplated for a size 6 femoral component, an image and/or video capturesystem integrated or attached to the OHMD that the surgeon is wearingcan read the QR code as the surgeon takes the femoral component and asthe femoral component with the attached optical marker and QR codeenters the surgeon's field of view or enters the proximity of thesurgical field. The image and/or video capture system and related systemsoftware can read the QR code identifying that the implant component isof a size 4; the system software can then compare the information to thevirtual surgical plan of the patient or the templating and/or sizinginformation which can indicate that a size 6 femoral component wasplanned, then triggering an alert that an incorrect femoral componenthas entered the field of view of the surgeon or has entered into theproximity of the surgical field, as for example demarcated by anotheroptical marker. The alert can assist the surgeon in correcting the errorby switching to the correct size component.

An image and/or video capture system and/or a bar code and/or QR codereader integrated into, attached to or separate from the OHMD can alsobe used to read embedded information on the virtual surgical instrumentsand/or implant components for inventory management and billing andinvoicing purposes. For example, the image and/or video capture systemand/or a bar code and/or QR code reader can detect which instrumentswere used, monitor their frequency of use, and when a certainrecommended frequency of used has been reached, the system can triggeran alert to send the instrument for servicing. In some embodiments, theimage and/or video capture system and/or a bar code and/or QR codereader can detect which instruments were used and trigger an alert tosend the instruments used for sterilization. In some embodiments, theimage and/or video capture system and/or a bar code and/or QR codereader can detect which disposable instruments were used and trigger analert in the system to replenish the supply and send new, additionaldisposable instruments to replace the ones used. In some embodiments,the image and/or video capture system and/or a bar code and/or QR codereader can detect which implant components and other chargeablecomponents were used and trigger an alert in the system to replenish thesupply and send new, additional implant to replace the ones used; thealert can also trigger a command to generate an invoice to the hospitaland/or surgery center and to monitor payment.

Any of the foregoing embodiments can be applied to any surgical step andany surgical instrument or implant component during any type of surgery,e.g. knee replacement, hip replacement, shoulder replacement, ligamentrepair including ACL repair, spinal surgery, spinal fusion, e.g.anterior and posterior, vertebroplasty and/or kyphoplasty.

In some embodiments of the invention, pins or other implantable orattachable markers or calibration or registration phantoms or devicesincluding optical markers can be placed initially, for example in a boneor an osteophyte or bone spur or other bony anatomy or deformity.Registration of virtual image data, for example using anatomic landmarksor locations or an osteophyte or bone spur or other bony anatomy ordeformity, where the pins have been physically placed and optionallymarking those on an electronic image, and live patient data can beperformed. The pins can be optionally removed then, for example if theywould interfere with a step of the surgical procedure. After the step ofthe surgical procedure has been performed, e.g. a bone cut, the pins canoptionally be re-inserted into the pin holes remaining in the residualbone underneath the bone cut and the pins can be used for registered thevirtual data of the patient with the live data of the patient eventhough the surgical site and anatomy has been altered by the surgicalprocedure.

In some embodiments of the invention, the registration of virtualpatient data and live patient data using the techniques described hereincan be repeated after one or more surgical steps have been performed. Inthis case, the surgically altered tissue or tissue surface or tissuecontour or tissue perimeter or tissue volume or other tissue features inthe live patient can be matched to, superimposed onto and/or registeredwith the surgically altered tissue or tissue surface or tissue contouror tissue perimeter or tissue volume or other tissue features in thevirtual data of the patient, e.g. in a virtual surgical plan developedfor the patient. The matching, superimposing and/or registering of thelive data of the patient and the virtual data of the patient after thesurgical tissue alteration can be performed using the same techniquesdescribed in the foregoing or any of the other registration techniquesdescribed in the specification or any other registration technique knownin the art.

Registration of Virtual Patient Data and Live Patient Data Using PatientSpecific Markers or Templates

Various techniques have been described for registering virtual patientdata with live patient data using patient specific markers or templatesincluding those described in WO9325157A1, which is expresslyincorporated by reference herein.

In some embodiments of the invention, pre-operative imaging is performedto acquire 3D data of the patient. The pre-operative imaging can, forexample, entail ultrasound, CT or MRI, any of the foregoing, optionallywith administration of a contrast agent.

The pre-operative imaging can include a single area or region, such as alumbar spine or portions of a lumbar spine or one or more spinalsegments, or a single joint, such as a knee joint, hip joint, anklejoint, shoulder joint, elbow joint or wrist joint. Alternatively, thepre-operative imaging can include scanning through portions or all ofone or more adjacent joints. This approach can be beneficial wheninformation about a length of an extremity or axis alignment orrotational alignment is desirable. For example, in planning a hipreplacement surgery, it can be beneficial to have image informationthrough the distal femur and, optionally, the knee joint and/or theankle joint available to determine, for example, leg length. In planninga knee replacement surgery, it can be beneficial to have imageinformation through the hip joint and the ankle joint available. In thismanner, the center of the hip and the ankle joint can be, for example,determined. This information can be used to determine the mechanicalaxis alignment of the patient and, optionally, to plan for anymechanical axis correction.

The pre-operative imaging can also entail imaging in one or morepositions, e.g. prone, supine, upright, flexion, extension, lateralbending. Data obtained from scans with the patient in differentpositions can optionally be combined or fused. For example, an uprightstanding weight-bearing partial or full leg x-ray can be used todetermine the mechanical axis alignment of the leg. 3D data of the knee,e.g. from CT or MRI can be used to obtain detailed anatomic informationabout the joint, for example to derive a surface shape and to design apatient specific marker or template. The information from the uprightscan can be used to align the patient specific marker or template oraspects of it in relationship to the mechanical axis. The informationfrom the 3D knee scan can be used to derive one or more patient specificsurfaces that fit to the unique shape of the patient.

In a patient with spinal symptoms, 3D data of the spine can be obtained,for example, with a CT or MRI scan or a rotational fluoroscopy or C-armscan. Upright imaging, for example in flexion and extension, can be usedto determine the presence and degree of spinal instability, for exampleprior to an intended spinal fusion surgery with pedicle screws and/orcages. The degree of instability or slippage can be determined and usedto decide on the degree of intended correction, if any, or the degree ofa required foraminotomy, both of which can be optionally planned on the3D data. Lateral bending views can optionally be used to determine thedegree and angle of a partial vertebral corpectomy and the desiredplacement and/or height of intervertebral cages. Thus, data from uprightimaging studies can be combined or optionally fused with data fromsupine or prone imaging studies. Data from 2D imaging studies can becombined or fused with data from 3D imaging studies. The 3D data can beused to derive one or more patient specific surfaces that fit to theunique shape of the patient, e.g. to the unique shape of one or more ofthe patient's spinous processes, one or more of the patient's transverseprocesses, one or more of the patient's laminae, one or more of thepatient's articular processes, one or more of the patient's vertebralbody.

The patient specific marker or template can include one or more surfacesthat are designed and manufactured to fit the corresponding surface ofthe patient, typically like a negative or substantially a negative.Optional smoothing of the surface can be performed. Alternatively, thesurface can be intentionally “roughened” to include more surfacefeatures than the segment 3D surface of the patient's target anatomy.Such surface features can, for example, include spike or pin-likestructures to allow for enhanced fixation of the patient specific markeror template on the patient's tissue surface.

The patient specific marker or template can be developed from CT, MRI orultrasound scans as well as x-ray imaging. Principally, any multi-planar2D or 3D imaging modality is applicable, in particular when it providesinformation on surface shape or provides information to derive estimatesof surface shape of an anatomic region. The patient specific marker ortemplate can include one or more surfaces that are designed ormanufactured to fit in any joint or in a spine or other anatomiclocations a corresponding

-   -   Cartilage surface of a patient    -   Subchondral bone surface of a patient    -   Cortical bone surface of a patient    -   Osteophyte or bone spur of a patient    -   Bone defect of a patient    -   Exuberant bone formation of a patient    -   Subchondral cyst of a patient    -   Soft-tissue shape, e.g. the shape of a thigh or calf or lower        back, or thoracic region, or neck region, or foot or ankle        region, or shoulder region    -   Soft-tissue shape in different body poses or positions, e.g. in        prone position or in supine position or in lateral position    -   Ligament of a patient    -   Labrum of a patient    -   Meniscus of a patient    -   Organ shape of a patient    -   Organ rim or edge of a patient, e.g. a liver edge or spleen edge

Different imaging tests can be particularly amenable for a given tissue.For example, if the patient specific marker or template is designed tofit the cartilage shape of the patient, MRI and ultrasound or CTarthrography are ideally suited to provide the surface information. Ifthe patient specific marker or template is intended to fit thesubchondral bone shape or cortical bone shape, CT can be used, althoughMRI and ultrasound can also provide information on bone shape.

Patient specific markers or templates can be manufactured usingdifferent materials, e.g. ABS or nylon or different types of plastics ormetals. They can be machined, e.g. from a blank, wherein a CAD/CAMprocess transfers the patient specific shape information into themilling machines. They can also be produced using stereolithography or3D printing techniques known in the art. If 3D printing is used, anyresidual powder can be removed using an air cleaning operation and/or awater bath. 3D printing can be performed using powder based or liquidresin based approaches, including, but not limited to continuous liquidinterface production.

Patient specific markers or templates can include or incorporate opticalmarkers, e.g. optical markers with different geometric shapes orpatterns, with QR codes, with bar codes, with alphanumeric codes.Optionally, geometric shapes or patterns, QR codes, bar codes,alphanumeric codes can be printed, for example when 3D printing is usedfor manufacturing patient specific markers or templates. 3D printing canbe performed with software, e.g.

Materialise Magics (Materialise, Leuven, Belgium), and hardware known inthe art, e.g. 3D printers from 3D Systems, Rock Hill, S.C., or ConceptLaser, Lichtenfels, Germany.

Patient specific markers or templates can be made with differentmaterial properties. For example, they can be non-elastic, semi-elasticor elastic. They can be hard. They can be solid or include hollow spacesor openings. They can be opaque. Patient specific markers or templatescan be semi-opaque. Patient specific markers can be transparent. In someembodiments, a patient specific marker or template can be semi-opaque orsemi-transparent. However, when the patient specific marker or templatescomes in contact with the patient and the patient specific surface(s) ofthe marker or template achieves a good fit with the correspondingsurface of the patient, the patient specific marker or template becomestransparent due to the tissue moisture on the corresponding surface ofthe patient.

One or more patient specific markers or templates can be used on a firstsurface of a joint. One or more patient specific markers can be used ona second surface of a joint. The first and second surface can be on thesame weight-bearing side of the joint. The first and second surface canbe on opposite sides of the joint. The one or more patient specificmarkers or templates on the first surface of the joint cannot beconnected to the one or more patient specific markers or templates onthe second surface of the joint. In some embodiments, the one or morepatient specific markers or templates on the first surface of the jointcan, optionally, be connected or linked to the second surface of thejoint. Thus, one or more patient specific markers or templates canoptionally be cross-referenced.

Patient specific markers or templates can be designed for any joint, anyportion of a spine, and any tissue of the human body. Patient specificmarkers or templates typically include one or more surfaces or shapesdesigned to fit a corresponding surface or shape of a patient.Representative, non-limiting examples of patient surfaces to whichpatient specific markers or templates can be designed and/or fittedinclude:

Spine:

-   -   A portion or an entire spinous process    -   A portion or an entire spinal lamina    -   A portion or an entire spinal articular process    -   A portion of or an entire facet joint    -   A portion of or an entire transverse process    -   A portion of or an entire pedicle    -   A portion of or an entire vertebral body    -   A portion of or an entire intervertebral disk    -   A portion of or an entire spinal osteophyte    -   A portion of or an entire spinal bone spur    -   A portion of or an entire spinal fracture    -   A portion of or an entire vertebral body fracture    -   Combinations of any of the foregoing

Hip:

-   -   A portion of or an entire acetabulum    -   A portion of or an entire edge of an acetabulum    -   Multiple portions of an edge of an acetabulum    -   A portion of an iliac wall    -   A portion of a pubic bone    -   A portion of an ischial bone    -   A portion of or an entire greater trochanter    -   A portion of or an entire lesser trochanter    -   A portion of or an entire femoral shaft    -   A portion of or an entire femoral neck    -   A portion of or an entire femoral head    -   A fovea capitis    -   A transverse acetabular ligament    -   A pulvinar    -   A ligamentum teres    -   A labrum    -   One or more osteophytes, femoral and/or acetabular    -   Combinations of any of the foregoing

Knee:

-   -   A portion or an entire medial femoral condyle    -   A portion or an entire lateral femoral condyle    -   A portion or an entire femoral notch    -   A portion or an entire trochlea    -   A portion of an anterior cortex of the femur    -   A portion of an anterior cortex of the femur with adjacent        portions of the trochlea    -   A portion of an anterior cortex of the femur with adjacent        portions of the trochlea and osteophytes when present    -   One or more osteophytes femoral and/or tibial    -   One or more bone spurs femoral and/or tibial    -   An epicondylar eminence    -   A portion or an entire medial tibial plateau    -   A portion or an entire lateral tibial plateau    -   A portion or an entire medial tibial spine    -   A portion or an entire lateral tibial spine    -   A portion of an anterior cortex of the tibia    -   A portion of an anterior cortex of the tibia and a portion of a        tibial plateau, medially or laterally or both    -   A portion of an anterior cortex of the tibia and a portion of a        tibial plateau, medially or laterally or both and osteophytes        when present    -   A portion or an entire patella    -   A medial edge of a patella    -   A lateral edge of a patella    -   A superior pole of a patella    -   An inferior pole of a patella    -   A patellar osteophyte    -   An anterior cruciate ligament    -   A posterior cruciate ligament    -   A medial collateral ligament    -   A lateral collateral ligament    -   A portion or an entire medial meniscus    -   A portion or an entire lateral meniscus    -   Combinations of any of the foregoing

Shoulder:

-   -   A portion or an entire glenoid    -   A portion or an entire coracoid process    -   A portion or an entire acromion    -   A portion of a clavicle    -   A portion or an entire humeral head    -   A portion or an entire humeral neck    -   A portion of a humeral shaft    -   One or more humeral osteophytes    -   One or more glenoid osteophytes    -   A portion or an entire glenoid labrum    -   A portion or an entire shoulder ligament, e.g. a coracoacromial        ligament, a superior, middle, or inferior glenohumeral ligament    -   A portion of a shoulder capsule    -   Combinations of any of the foregoing

Skull and Brain:

-   -   A portion of a calvarium    -   A portion of an occiput    -   A portion of a temporal bone    -   A portion of an occipital bone    -   A portion of a parietal bone    -   A portion of a frontal bone    -   A portion of a facial bone    -   A portion or an entire bony structure inside the skull    -   Portions or all of select gyri    -   Portions or all of select sulci    -   A portion of a sinus    -   A portion of a venous sinus    -   A portion of a vessel

Organs:

-   -   A portion of an organ, e.g. a superior pole or inferior pole of        a kidney    -   An edge or a margin of a liver, a spleen, a lung    -   A portion of a hepatic lobe    -   A portion of a vessel    -   A portion of a hiatus, e.g. in the liver or spleen    -   A portion of a uterus

The patient specific marker or template can be designed or fitted to anyof the previously mentioned tissues, if applicable for a particularanatomic region, e.g. cartilage, subchondral bone, cortical bone,osteophytes etc. The patient specific marker or template can be designedor fitted to normal tissue only. The patient specific marker or templatecan be designed or fitted to abnormal or diseased tissue only. Thepatient specific marker or template can be designed or fitted tocombinations of normal and abnormal or diseased tissue. For example, thepatient specific marker can be designed to normal cartilage, or todiseased cartilage, or to combinations of normal and diseased cartilage,e.g. on the same or opposing joint surfaces. Patient specific markerscan be used to register one or more normal or pathologic tissues orstructures in a common coordinate system, for example with one or moreOHMDs and virtual data of the patient. Virtual and physical surgicalinstruments and implant components can also be registered in the commoncoordinate system.

The patient specific marker or template can be designed using virtualdata of the patient, e.g. from a pre-operative imaging study such as aCT scan, MRI scan or ultrasound scan. The patient specific marker ortemplate includes one or more surfaces that are designed and/ormanufacture to achieve a close fit with a corresponding surface of thepatient.

In some embodiments of the invention, a surgeon or an operator can applythe patient specific marker or template to the corresponding tissue ofthe patient. Once a satisfactory fit has been achieved and the twocorresponding surfaces are substantially in contact, the patientspecific marker or template can be used to register the virtual data ofthe patient and an optional virtual surgical plan with the live data ofthe patient. By applying the patient specific marker or template to itscorresponding surface(s) on the patient, the surgeon is effectivelyidentifying corresponding structures or surfaces in the virtual data andthe live data of the patient.

The position, location and/or orientation of the patient specific markeror template can then be determined in relationship to the OHMD. Any ofthe embodiments described herein can be applied for determining theposition, location and/or orientation of the patient specific marker ortemplate in relationship to the OHMD. For example, the side of thepatient specific marker or template that is opposite the patientspecific surface can include certain standardized geometric features,e.g. rectangles, triangles, circles and the like, that can be readilyrecognized by an image and/or video capture system integrated into orattached to or coupled to the OHMD. In alternative embodiments, thepatient specific marker or template can include one or more IMUs,including, for example, accelerometers, magnetometers, and gyroscopes,similar, for example, to the OHMD. In some embodiments, the patientspecific marker or template can include one or more radiofrequency tagsor markers or retroreflective markers and its position, location and/ororientation can be captured by a surgical navigation system.Radiofrequency tags can be active or passive. Optionally, the OHMD mayalso include one or more radiofrequency tags or markers orretroreflective markers and its position, location and/or orientationcan also be captured by the surgical navigation system andcross-referenced to the patient specific marker or template. The patientspecific marker or template can also include light sources, such aslasers or LEDs. A laser can be projected, for example, on a wall or aceiling and the OHMD can be referenced in relationship to that. An LEDattached to or integrated into the patient specific marker or templatecan be recognized, for example, by an image and/or video capture systemintegrated into or attached to r coupled to the OHMD.

In an additional embodiment, one or more of the surgical instrumentsand/or one or more of the implantable devices used during the surgerycan include can include certain standardized geometric features, e.g.rectangles, triangles, circles and the like, that can be readilyrecognized by an image and/or video capture system integrated into orattached to or coupled to the OHMD. In alternative embodiments, one ormore of the surgical instruments and/or one or more of the implantabledevices used during the surgery can include one or more IMUs, including,for example, accelerometers, magnetometers, and gyroscopes, similar, forexample, to the OHMD. In some embodiments, one or more of the surgicalinstruments and/or one or more of the implantable devices used duringthe surgery can include one or more radiofrequency tags or markers orretroreflective markers and its position, location and/or orientationcan be captured by a surgical navigation system. Optionally, the OHMDmay also include one or more radiofrequency tags or markers orretroreflective markers and its position, location and/or orientationcan also be captured by the surgical navigation system andcross-referenced to the patient specific marker or template and/or theone or more of the surgical instruments and/or one or more of theimplantable devices used during the surgery. One or more of the surgicalinstruments and/or one or more of the implantable devices used duringthe surgery can also include light sources, such as lasers or LEDs. Alaser can be projected, for example, on a wall or a ceiling and the OHMDand the patient can be referenced in relationship to that. An LEDattached to or integrated into the one or more of the surgicalinstruments and/or one or more of the implantable devices used duringthe surgery can be recognized, for example, by an image and/or videocapture system integrated into or attached to or coupled to the OHMD.Optionally, multiple LEDs can be used. Optionally, two or more of themultiple LEDs emit light with different wavelength or color. The two ormore LEDs can be located in spatially defined locations andorientations, e.g. at a pre-defined or fixed distance and at one or morepre-defined or fixed angles. In this manner, the two or more LEDs can belocated by an image and/or video capture system integrated into,attached to or separate from the OHMD and their measured distance and/orangles as seen through the image and/or video capture system can, forexample, be used to determine the distance and or orientation of theoperator to the target anatomy, e.g. when the image and/or video capturesystem is close to the operator's eyes. By using LEDs with differentwavelength or color, the image and/or video capture system candifferentiate between different LEDs; when the LEDs are arranged in aknown spatial orientation, this information can be helpful forincreasing the accuracy of the registration and/or for obtainingaccurate distance, angle, direction and/or velocity measurements. Theuse of two or more LEDs with different wavelength and color andmeasurements or registration as described above are applicablethroughout the specification in all embodiments that incorporate the useof LEDs or that are amenable to using LEDs.

Optionally, the patient specific marker or template and, optionally, oneor more of the surgical instruments and/or one or more of theimplantable devices used during the surgery can also include colormarkings, optionally with different geometric shapes or located ororiented at different, known locations and different, known angles, thatcan be used, for example, by an image and/or video capture systemintegrated into or attached to or coupled to an OHMD to recognize suchpatterns and, for example, to estimate distances and angles, e.g. fromthe surgical site to the OHMD, or distances and angles between twomarkings, two surgical instruments or medical device components.

Optionally, the patient specific marker or template and, optionally, oneor more of the surgical instruments and/or one or more of theimplantable devices used during the surgery can also include scales,e.g. of metric distances, inches, or angles that can be used, forexample, by an image and/or video capture system integrated into orattached to or coupled to an OHMD to recognize such scales or anglesand, for example, to estimate distances and angles, e.g. from thesurgical site to the OHMD, or distances and angles between two surgicalinstruments or medical device components.

In some embodiments of the invention, the patient specific marker ortemplate can be attached to the corresponding surface of the patient orto an adjacent surface of the patient, for example using tissue gluesuch as fibrin glue or a pin or a staple.

In some embodiments, the patient specific marker or template can includeopenings or guides, for example for accepting a surgical instrument ortool such as a bur, a saw, a reamer, a pin, a screw and any otherinstrument or tool known in the art.

By cross-referencing virtual patient data and live patient data with useof a patient specific marker or template and, optionally, one or more ofthe surgical instruments and/or one or more of the implantable devicesused during the surgery and an OHMD, any coordinate information,distance information, axis information, functional information containedin the virtual patient data can now be available and used during thesurgery.

In some embodiments of the invention, the registration of virtualpatient data and live patient data using the techniques described hereincan be repeated after one or more surgical steps have been performed. Inthis case, the surgically altered tissue or tissue surface or tissuecontour or tissue perimeter or tissue volume or other tissue features inthe live patient can be matched to, superimposed onto and/or registeredwith the surgically altered tissue or tissue surface or tissue contouror tissue perimeter or tissue volume or other tissue features in thevirtual data of the patient, e.g. in a virtual surgical plan developedfor the patient. The matching, superimposing and/or registering of thelive data of the patient and the virtual data of the patient after thesurgical tissue alteration can be performed using the same techniquesdescribed in the foregoing or any of the other registration techniquesdescribed in the specification or any other registration technique knownin the art.

Registration of Virtual Patient Data and Live Patient Data UsingIntraoperative Imaging

In some embodiments of the invention, intraoperative imaging, forexample using x-ray imaging or CT imaging and/or ultrasound imaging, canbe performed. Virtual patient data obtained intraoperatively usingintraoperative imaging can be used to register virtual patient dataobtained preoperatively, for example using preoperative x-ray,ultrasound, CT or MRI imaging. The registration of preoperative andintraoperative virtual data of the patient and live data of the patientin a common coordinate system with one or more OHMDs can be performed,for example, by identifying and, optionally, marking correspondinglandmarks, surfaces, object shapes, e.g. of a surgical site or targettissue, in the preoperative virtual data of the patient, theintraoperative virtual data of the patient, e.g. on electronic 2D or 3Dimages of one or more of the foregoing, and the live data of thepatient. Virtual preoperative, virtual intraoperative and live data caninclude an osteophyte or bone spur or other bony anatomy or deformity.Virtual and physical surgical instruments and implant components canalso be registered in the common coordinate system.

This embodiment can be advantageous when the amount of informationobtained with intraoperative imaging is, for example, anatomically or inother ways more limited than the amount of information available withpreoperative imaging or vice versa.

For example, intraoperative imaging may be performed using x-rayimaging, which is commonly only two-dimensional in nature. X-ray imagingcan be augmented through image acquisition in more than one plane, e.g.orthogonal planes or one or more planes separated by a defined angle.Intraoperative x-ray images can be used to identify certain landmarks orshapes that can then be registered to preoperative imaging and/or livedata of the patient during surgery. Preoperative imaging can,optionally, include 3D image data, for example obtained with CT or MRI.Acquisition of intraoperative images in multiple planes can be helpfulto more accurately define the location of certain landmarks, contours orshapes intended for use in a registration of preoperative virtual data,intraoperative virtual data and live data of the patient. For purposesof clarification, intraoperative virtual data of the patient can beintraoperative images of the patient in 2D or 3D.

For example, in a spinal procedure such as vertebroplasty, kyphoplasty,pedicle screw placement, or placement of anterior spinal deviceincluding artificial disks or cages, intraoperative x-ray imaging can beused to identify, for example, the spinal level targeted for thesurgery, in an AP projection certain landmarks or contours, e.g. the tipof a spinous process, a facet joint, the superior or inferior tip of afacet joint, the cortical edge of a lamina, a superior or inferiorendplate or an osteophyte or bone spur or other bony anatomy ordeformity. Optionally, the distance of the x-ray tube from the patientresulting in x-ray magnification can be factored into any registrationin order to improve the accuracy of the registration of virtualpreoperative data of the patient and virtual intraoperative data of thepatient or live data of the patient. The intraoperative x-ray images canthen be registered and, optionally, superimposed onto the preoperativedata of the patient or the live data of the patient in the projection bythe OHMD. The intraoperative virtual data of the patient, e.g. the tipof a spinous process, a facet joint, the superior or inferior tip of afacet joint, the cortical edge of a lamina, a superior or inferiorendplate, can be registered to the live data of the patient, for exampleby touching the corresponding anatomic landmarks with a pointing deviceor a needle or a pin inserted through the skin and by cross-referencingthe location of the tip of the live data pointing device with theintraoperative virtual data of the patient. In this manner, any one ofpreoperative virtual data of the patient, intraoperative virtual data ofthe patient, and live data of the patient and combinations thereof canbe co-registered. Two or three of these data sets, preoperative virtualdata of the patient, intraoperative virtual data of the patient, andlive data of the patient, can optionally be seen in the OHMD. However,in many embodiments, intraoperative imaging may only be used forenhancing the accuracy of the registration of preoperative virtual dataof the patient and live data of the patient and, for example,preoperative virtual data of the patient and/or a medical deviceintended for placement in a surgical site will be displayed by the OHMDtogether with the view of the live data of the patient or the surgicalsite.

In some embodiments of the invention, the registration of virtualpatient data and live patient data using the techniques described hereincan be repeated after one or more surgical steps have been performedand, optionally, intraoperative imaging can be repeated. In this case,the surgically altered tissue or tissue surface or tissue contour ortissue perimeter or tissue volume or other tissue features in the livepatient or in the intraoperative repeat imaging data of the patient canbe matched to, superimposed onto and/or registered with the surgicallyaltered tissue or tissue surface or tissue contour or tissue perimeteror tissue volume or other tissue features in the virtual data of thepatient, e.g. in a virtual surgical plan developed for the patient. Thematching, superimposing and/or registering of the live data of thepatient and the virtual data of the patient after the surgical tissuealteration can be performed using the same techniques described in theforegoing or any of the other registration techniques described in thespecification or any other registration technique known in the art.

Registration of Virtual Patient Data and Live Patient Data Using SkinMarkers or Soft-Tissue Markers

In some embodiments of the invention, skin markers and soft-tissuemarkers, calibration or registration phantoms or devices can be used forregistering preoperative virtual data, optionally intraoperative virtualdata such as data obtained from intraoperative x-ray imaging, and livedata seen through the OHMD in a common coordinate system with one ormore OHMDs. Virtual and physical surgical instruments and implantcomponents can also be registered in the common coordinate system. Forexample, an initial registration between preoperative virtual data andlive data of the patient can happen at the beginning of the procedure.The initial registration can, for example, be performed usingcorresponding anatomic landmarks, surfaces or shapes, or usingintraoperative imaging resulting in intraoperative virtual data or anyof the other embodiments described in this invention. The registrationcan be used, for example, to place the virtual data and the live dataand the optical head mounted display into a common coordinate system.Skin markers, calibration or registration phantoms or devices can thenbe applied. Virtual and physical surgical instruments and implantcomponents can also be registered in the common coordinate system.Alternatively, or in addition, soft-tissue markers, calibration orregistration phantoms or devices can be applied. Typically, more thanone, such as two, three, four or more skin markers and soft-tissuemarkers, calibration or registration phantoms or devices will beapplied. For clarity, the terms implantable markers, attachable markers,skin markers, soft-tissue markers, calibration or registration phantomsor devices as used through the application can include optical markers,e.g. optical markers with different geometric shapes or patterns, withQR codes, with bar codes, with alphanumeric codes. Skin markers andsoft-tissue markers, calibration or registration phantoms or devicescan, for example, be applied to the skin or the soft-tissue using a formof tissue compatible adhesive, including fibrin glue and the like. Insome embodiments, one, two, three, four or more skin markers andsoft-tissue markers, calibration or registration phantoms or devices canbe included in a surgical drape or dressing or a transparent filmapplied to the skin prior to the procedure. The skin markers andsoft-tissue markers, calibration or registration phantoms or devices canthen be registered in the live data and cross-referenced to virtualdata. The skin markers and soft-tissue markers, calibration orregistration phantoms or devices can subsequently used, for example,when the surgical site is altered and the landmarks, surface or shapethat was used for the initial registration of virtual and live data havebeen altered or removed and cannot be used or cannot be used reliablyfor maintaining registration between virtual data and live data. Virtualpreoperative, virtual intraoperative and live data can include anosteophyte or bone spur or other bony anatomy or deformity.

In some embodiments of the invention, the registration of virtualpatient data and live patient data using the techniques described hereincan be repeated after one or more surgical steps have been performed. Inthis case, the surgically altered tissue or tissue surface or tissuecontour or tissue perimeter or tissue volume or other tissue features inthe live patient can be matched to, superimposed onto and/or registeredwith the surgically altered tissue or tissue surface or tissue contouror tissue perimeter or tissue volume or other tissue features in thevirtual data of the patient, e.g. in a virtual surgical plan developedfor the patient. The matching, superimposing and/or registering of thelive data of the patient and the virtual data of the patient after thesurgical tissue alteration can be performed using the same techniquesdescribed in the foregoing or any of the other registration techniquesdescribed in the specification or any other registration technique knownin the art.

The same skin markers or soft-tissue markers or calibration phantoms orregistration phantoms can be used after one or more surgical steps havebeen performed if the markers or phantoms are still in place.Alternatively, re-registration of the live data of the patient andvirtual data of the patient can be performed after one or more surgicalsteps or surgical alterations. Following re-registration, one or morenew skin markers or soft-tissue markers or calibration phantoms orregistration phantoms can be applied and cross-referenced to there-registered live and virtual data after the surgical step oralteration. The skin markers or soft-tissue markers or calibrationphantoms or registration phantoms can then be used for subsequentmatching, superimposition, movement and registration of live patientdata and virtual patient data.

Registration of Virtual Patient Data and Live Patient Data UsingCalibration or Registration Phantoms with Defined Dimensions or Shapes

In some embodiments of the invention, calibration or registrationphantoms with defined dimensions or shapes can be used to perform theregistration of virtual data of the patient and live data of thepatient. The calibration or registration phantoms can be of primarilytwo-dimensional or three-dimensional nature. For example, a calibrationor registration phantom can be arranged or located primarily in a singleplane. Other calibration phantoms can be located in multiple planes,thereby creating the opportunity for registration using more than oneplanes. For clarity, the terms calibration or registration phantoms,implantable markers, attachable markers, skin markers, soft-tissuemarkers, or devices as used through the application can include opticalmarkers, e.g. optical markers with different geometric shapes orpatterns, with QR codes, with bar codes, with alphanumeric codes.

Such calibration or registration phantoms can be, for example, attachedto the patient's skin. The calibration or registration phantom can beintegrated or attached to a surgical drape. The calibration orregistration phantom can be attached to the patient's tissue. Thecalibration or registration phantom can be part of or a component of amedical device. The part or component of the medical device willtypically have known dimensions. By using calibration or registrationphantoms, as well as other markers, the live data of a patient and thevirtual data of the patient can be registered in a common coordinatesystem, for example with one or more OHMDs. Virtual and physicalsurgical instruments and implant components can also be registered inthe common coordinate system.

In some embodiments, the calibration or registration phantom includesknown dimensions, angles or geometric 2D or 3D shapes. For example, thecalibration or registration phantom can include structures such as

-   -   circles, ovoids, ellipses, squares, rectangles, complex 2D        geometries, 2D geometries with one or more defined distances, 2D        geometries with one or more defined angles    -   spheres, egg shaped structures, cylinders, cubes, cuboids,        complex 3D geometries or shapes, 3D geometries with one or more        defined distances, 3D geometries with one or more defined        angles, 3D geometries with one or more defined surfaces

Optionally, the calibration or registration phantoms can be radiopaqueif pre-operative or intra-operative imaging is performed using animaging modality with ionizing radiation, e.g. x-ray imaging,fluoroscopy in 2D or 3D, CT, cone beam CT etc.

In some embodiments, the calibration or registration phantom can be MRIvisible or nuclear scintigraphy or SPECT visible or PET visible, forexample by including portions or containers in the phantom containingGadolinium-DTPA doped or radionuclide doped or PET isotope emittingwater. Any contrast agent or MRI or nuclear scintigraphy or SPECT or PETvisible agent known in the art can be used in this fashion.

In some embodiments, the calibration or registration phantom includesretroreflective markers or features which facilitate detection by animage and/or video capture system. The calibration or registrationphantom can also be highlighted against the patient's tissue(s)including blood as well as surgical drapes through a choice of selectcolors, e.g. a bright green, bright blue, bright yellow, bright pinketc. Color combinations are possible. Any color or color combinationknown in the art can be used.

The calibration or registration phantom can optionally include LEDs,optionally battery powered. More than one LED can be used. The LEDs canemit a light of a known color, hue and intensity, preferably selected tobe readily identifiable by the image and/or video capture system and anysegmentation techniques or algorithms used for detecting the location,position and/or orientation of the LEDs.

The LEDs can be arranged in a spatially defined way, with two or moreLEDs arranged at a defined distance or distances, at a defined angle orangles, in substantially the same plane or different planes. If LEDs arearranged in different planes, the spatial orientation of the planes isfor example known and defined.

When two or more LEDs are used, the two or more LEDs can emit lightutilizing different wavelengths, colors, intensity and, optionally also,blinking frequency. In this manner, an image and/or video capture systemintegrated into, attached to or separate from the OHMD can recognizeeach different LED based on one or more of their different wavelength,color, intensity and/or blinking frequency. When the LEDs are arrange ina spatially defined and known manner, e.g. using known distances orangles within the same plane or different planes, the identification ofeach individual LED and the change in distances and angles measured bythe image and/or video capture system can be used to determine theposition, location and/or orientation of the OHMD and/or the operator'shead (e.g. if the image and/or video capture system is integrated intothe OHMD or attached to the OHMD) or, in some applications, the movementof the patient or body part to which the calibration or registrationphantom and LEDs are attached.

LEDs used throughout the specification can be re-useable. LEDs usedthroughout the specification can also be disposable, optionally withintegrated, disposable battery cells/batteries. LEDs can be operatedutilizing wires, e.g. connected to a power supply and/or connected to awired user interface or control unit. LEDs can be wireless, e.g. withoutattached power supply (e.g. battery operated) and/or connected to awireless (e.g. WiFi, Bluetooth) control unit.

LEDs can be connected and/or organized in LIF networks. One or more LIFnetworks can be used, for example, to transmit or receive data orinformation back and forth from the one or more OHMDs to a control unitor computer, optionally with a user interface. In this example, LEDsparticipating or connected in the one or more LIF networks can beintegrated into or attached to the OHMD. LEDs participating or connectedin the one or more LIF networks can be attached to or, when applicable,integrated into any location or site on the surgeon, the OR staff, thepatient, the surgical site, one or more OHMDs, one or more navigationsystems, one or more navigation markers, e.g. retroreflective markers,infrared markers, RF markers; one or more optical markers, calibrationor registration phantoms.

An LIF network can also be used to transmit or receive data orinformation about the spatial position, orientation, direction ofmovement, speed of movement etc. of individual LEDs. The same LEDs whoserelative position, orientation, direction of movement, speed ofmovement, e.g. in relationship to the surgeon or the patient or thesurgical site, is being measured, e.g. using an image and/or videocapture system, can be used to transmit or receive information in theLIF network, optionally using different wavelengths, color, frequency,blinking patterns depending on the type of data being transmitted. Theinformation can be about the position, orientation, direction ofmovement, speed of movement of individual LEDs. The information can alsobe data that are being transmitted or received by the OHMD. Theinformation can be the information or data that are being displayed bythe OHMD. The information can be information generated or received bynavigation markers, RF markers. The information can be informationcaptured by one or more image and/or video capture systems or cameras.

1, 2, 3, 4 or more LEDs can be connected to or attached to the patient,the target anatomy, the surgical site, the surgical site after a first,second or more surgical alterations, for example executed using avirtual surgical plan, the OHMD, a second, third and/or additionalOHMDs, for example worn by a second surgeon, a scrub nurse, other ORpersonnel, the hand, forearm, upper arm and or other body parts of thesurgeon/operator.

The relative position, orientation, movement, direction of movement,velocity of movement of each LED can be determined, for example usingone or more image and/or video capture systems, e.g. integrated into,attached to or separate from the one or more OHMDs, e.g. when the one ormore LEDs emit light utilizing different wavelengths, colors, intensityand, optionally also, blinking frequency.

The calibration or registration phantom can optionally include one ormore lasers, optionally battery powered. More than one laser can beused. The laser can emit a light of a known color, hue and intensity,for example selected to be readily identifiable by the image and/orvideo capture system and any segmentation techniques or algorithms usedfor detecting the location, position and/or orientation of the laser.

The laser can be arranged in a spatially defined way, with two or morelasers arranged at a defined distance or distances, at a defined angleor angles, in substantially the same plane or different planes. Iflasers are arranged in different planes, the spatial orientation of theplanes can be known and defined.

The calibration or registration phantom can optionally includeradiofrequency (RF) transmitters, optionally battery powered. More thanone RF transmitter can be used. The RF transmitters can transmit asignal or signals selected to be readily identifiable by an RF receiversystem used for detecting the location, position and/or orientation ofthe RF transmitters. One or more RF transmitters can transmit signalswith different frequency and intensity, thereby permittingdifferentiation of the different RF transmitters by the RF receiversystem.

The RF transmitters can be arranged in a spatially defined way, with twoor more RF transmitters arranged at a defined distance or distances, ata defined angle or angles, in substantially the same plane or differentplanes. If RF transmitters are arranged in different planes, the spatialorientation of the planes is can be known and defined.

The calibration or registration phantom can optionally includeultrasound (US) transmitters, optionally battery powered. More than oneUS transmitter can be used. The US transmitters can transmit a signal orsignals selected to be readily identifiable by an US receiver ortransducer system used for detecting the location, position and/ororientation of the US transmitters. One or more US transmitters cantransmit signal with different frequency and intensity, therebypermitting differentiation of the different US transmitters by the USreceiver or transducer system.

The US transmitters can be arranged in a spatially defined way, with twoor more US transmitters arranged at a defined distance or distances, ata defined angle or angles, in substantially the same plane or differentplanes. If US transmitters are arranged in different planes, the spatialorientation of the planes is can be known and defined.

Calibration phantoms or registration phantoms can be used forpre-operative imaging and/or for intraoperative imaging and/or imagecapture of live data, for example using an image and/or video capturesystem attached to or integrated into the OHMD or coupled to the OHMD orseparate from the OHMD. Virtual preoperative, virtual intraoperative andlive data can include an osteophyte or bone spur or other bony anatomyor deformity.

If the same calibration or registration phantom is used forpre-operative imaging and for intra-operative imaging, optionally, theimaging can be performed using the same imaging modality, e.g. x-rayimaging, and, for example, using the same orientation of the patient inrelationship to the x-ray source and the detector system and, forexample using the same distance of the patient in relationship to thex-ray source and the detector system. Using this approach, the anatomicstructures visualized on the pre-operative imaging and intra-operativeimaging can be superimposed and registered, optionally in the samecoordinate system.

In the event, the calibration or registration phantom has beenpositioned differently on the patient for the pre-operative imaging andfor the intraoperative imaging data acquisition, the difference inlocation or position or coordinates can be determined using theco-registration of the anatomic data visualized on the pre-operativeimaging and intra-operative imaging. An adjustment for the difference inphantom location from the pre-operative to the intraoperative data canbe performed; this adjustment can optionally be defined as a phantomoffset between pre-operative and intra-operative data. Virtualpreoperative, virtual intraoperative and live data can include anosteophyte or bone spur or other bony anatomy or deformity.

As an alternative to the anatomic registration from the anatomicstructures visualized on the pre-operative imaging and intra-operativeimaging, the registration between pre-operative imaging data andintra-operative live data visualized through the OHMD or an attached,integrated or separate image and/or video capture system can beperformed alternatively now using the calibration or registrationphantom as visualized or as identified optically during the surgery, forexample using the phantom offset between pre-operative andintra-operative data. In general, the initial registration of virtualdata and live data is possible using any of the techniques describedherein, e.g. using anatomic features, anatomic landmarks, intraoperativeimaging etc. Then co-registration of the calibration or registrationphantom, e.g. in the same coordinate system, can be performed. Ifinitial registration fails during the surgical procedure, registrationcan be maintained using the calibration or registration phantom. Forthis purpose, the position, location, orientation and/or alignment ofthe calibration or registration phantom will be continuously orintermittently monitored using an image and/or video capture system,which can be integrated into or attached to the OHMD or coupled to theOHMD or separate from the OHMD.

In some embodiments of the invention, the preoperative imaging canentail a cross-sectional imaging modality, e.g. computed tomography,which can optionally generate 3D data of the patient, e.g. in the formof a spiral or a helical CT scan and, optionally, a 3D reconstruction.The 3D data of the patient, e.g. the spiral or helical CT scan or 3Dreconstruction, can be re-projected into a 2D image, creating an x-raylike transmission image of the patient, e.g. of the bony structures ofthe patient including, but not limited to an osteophyte or bone spur orother bony anatomy or deformity. Optionally, this 2D re-projection ofthe 3D data, e.g. CT data, can be performed using the same plane orprojection or view angle and, for example, the same or similarmagnification as can be used subsequently during surgery with anintraoperative x-ray imaging test. The film-focus and, optionally,object distance of the x-ray system used for the intraoperative imagingpart can be known at the time of the re-projection of the preoperative3D data, so that the magnification of the patient or anatomic dataresulting for a given intraoperative film-focus and optionally objectdistance will be matched or reflected in the re-projected pre-operativedata. If the film-focus and, optionally, object distance of the x-raysystem used for the intraoperative imaging part is not known at the timeof the re-projection of the preoperative 3D data, the magnification ofthe re-projected data can be adjusted when they are visualized with andoptionally superimposed onto the 2D intraoperative imaging data of thepatient or anatomic data resulting for a given intraoperative film-focusand optionally object distance so that the magnification of bothre-projected and intraoperative imaging data will be matched orsubstantially similar. Such matching in magnification can be achieved,for example, by aligning certain features or anatomic landmarks orpathologic tissues including an osteophyte or bone spur or other bonyanatomy or deformity in the pre-operative re-projected data with theintraoperative data and adjusting the magnification until the feature orlandmarks are substantially superimposed or substantially matching. Withthis approach, pre-operative imaging data can use the benefit of 3D dataincluding, for example, more accurate three-dimensional placement of animplant component such as a spinal component or a component for jointreplacement or fracture repair. Similarly, certain anatomic landmarks orfeatures can be detected and utilized for surgical planning in the 3Ddata set. When the 3D data are then re-projected into a 2D re-projectionor view, anatomic landmarks, features or data or pathologic data can bereadily matched up or aligned with corresponding anatomic landmarks,features or data or pathologic data in the corresponding portions of theintraoperative 2D imaging study, e.g. intraoperative x-rays. Thus, whiledifferent 3D preoperative and 2D intraoperative imaging modalities canbe used, 2D re-projection allows for cross-referencing and, optionally,co-registration of the 2D and 3D data sets. Any 2D and 3D imagingmodality known in the art can be used in this manner.

In additional embodiments, the calibration/registration phantom can beused

-   -   1.) To estimate distance, position, orientation of OHMD from the        patient, for primary or back-up registration, for example used        in conjunction with an image and/or video capture system        integrated into, attached to or coupled to or separate from the        OHMD    -   2.) To estimate distance, position, orientation of target tissue        or surgical site underneath the patient's skin, e.g. after        cross-registration with pre-operative and/or intra-operative        imaging data    -   3.) To estimate the path of a surgical instrument or to estimate        the location of a desired implantation site for a medical device        or implant or transplant    -   4.) To update a surgical plan

The calibration or registration phantom can be used in physical timemode, using physical time registration, for example using an imageand/or video capture system integrated into, attached to, coupled to, orseparate from the OHMD, which can optionally operate in physical timemode. Physical time mode can, for example, mean that image capture isperformed with more than 5 frames/second, 10 frames/second, 15frames/second, 20 frames/second, 30 frames/second etc.

If images generated with the image and/or video capture system aresegmented or, for example, image processing or pattern recognition isperformed, this can optionally be performed on each frame generated withthe image and/or video capture system. Alternatively, segmentation orimage processing or pattern recognition can be performed on a subset ofthe image frames captured with the image and/or video capture system.Segmentation, image processing or pattern recognition data can beaveraged between frames. The foregoing embodiments are applicable to allembodiments in this specification that utilize image capture.

Image processing can be performed to include data from one or moreosteophytes or bone spurs or other bony anatomy or deformity. The one ormore osteophytes or bone spurs or other bony anatomy or deformity can beused for purposes of registration of virtual and live data, includingvirtual preoperative and virtual intraoperative imaging or virtualfunctional data. Image processing can also be performed to exclude datafrom one or more osteophytes or bone spurs or other bony anatomy ordeformity. The one or more osteophytes or bone spurs or other bonyanatomy or deformity can be excluded or omitted from any data used forpurposes of registration of virtual and live data, including virtualpreoperative and virtual intraoperative imaging or virtual functionaldata. The inclusion or exclusion of one or more osteophytes or bonespurs or other bony anatomy or deformity can be selected based on theanatomic site, the surgical site, and/or the desired accuracy of thesegmentation or the registration of virtual data and live data.

The calibration or registration phantom can be used in non-physical timemode, e.g. an intermittent mode, for example using an image and/or videocapture system integrated into, attached to, coupled to, or separatefrom the OHMD, which can optionally operate in intermittent mode.Intermittent mode use of the calibration or registration phantom can beperformed, for example, by using a timer or timing device, wherein imagecapture and registration is performed every 10 seconds, 8 seconds, 5seconds, 3 seconds, 2 seconds, 1 second etc.

In some embodiments, physical-time and intermittent registration usingthe calibration or registration phantom will be selected or designed sothat the data generated will for example not exceed the temporalresolution of the image and/or video capture system and/or the temporalresolution of the segmentation or image processing or patternrecognition used for the registration.

In any of the foregoing embodiments, the accuracy of registration canoptionally be improved by using multiple registration points, patterns,planes or surfaces. In general, the accuracy of registration willimprove with an increasing number of registration points, patterns,planes or surfaces. These may, in some embodiments, not exceed thespatial resolution of the image and/or video capture system. In someembodiments, these may exceed the spatial resolution of the image and/orvideo capture system. In that situation, optionally, down-sampling ofdata can be performed, e.g. by reducing the effective spatial resolutionin one, two or three planes or by reducing the spatial resolution inselect areas of the field of view seen through the OHMD or visualized inthe virtual data. Virtual preoperative, virtual intraoperative and livedata can include an osteophyte or bone spur or other bony anatomy ordeformity.

In some embodiments of the invention, the registration of virtualpatient data and live patient data using the techniques described hereincan be repeated after one or more surgical steps have been performed. Inthis case, the surgically altered tissue or tissue surface or tissuecontour or tissue perimeter or tissue volume or other tissue features inthe live patient can be matched to, superimposed onto and/or registeredwith the surgically altered tissue or tissue surface or tissue contouror tissue perimeter or tissue volume or other tissue features in thevirtual data of the patient, e.g. in a virtual surgical plan developedfor the patient. The matching, superimposing and/or registering of thelive data of the patient and the virtual data of the patient after thesurgical tissue alteration can be performed using the same techniquesdescribed in the foregoing or any of the other registration techniquesdescribed in the specification or any other registration technique knownin the art.

The same skin markers or soft-tissue markers or calibration phantoms orregistration phantoms can be used after one or more surgical steps havebeen performed if the markers or phantoms are still in place.Alternatively, re-registration of the live data of the patient andvirtual data of the patient can be performed after one or more surgicalsteps or surgical alterations. Following re-registration, one or morenew skin markers or soft-tissue markers or calibration phantoms orregistration phantoms can be applied and cross-referenced to there-registered live and virtual data after the surgical step oralteration. The skin markers or soft-tissue markers or calibrationphantoms or registration phantoms can then be used for subsequentmatching, superimposition, movement and registration of live patientdata and virtual patient data.

To Estimate Distance, Position, Orientation of OHMD from the Patient

If registration of virtual patient data and live patient data hasoccurred using any of the techniques or techniques described in thisspecification and if the calibration or registration phantom is alsoregistered in relationship to the live patient data, the calibration orregistration phantom can be used to maintain registration, for exampleon an intermittent or a physical-time basis, including while the surgeonor operator moves his or her head or body. The calibration orregistration phantom can, for example, not be moved during the surgery.If the calibration or registration phantom needs to be moved, it mayoptionally be re-registered in relationship to any live patient data,virtual patient data, pre-operative data and intra-operative data.

In this and related embodiments, the calibration or registration phantomwill be identified with regard to its location, position, orientation,alignment, surfaces or shape using an image and/or video capture systemand, optionally, segmentation, image processing or pattern recognitionand any other techniques known in the art for identifying an object inimage data. The image and/or video capture system can be integrated intoor attached to the OHMD. The image and/or video capture system can becoupled to or separate from the OHMD. The image and/or video capturesystem will be used to determine the location, position, orientation,alignment, surfaces or shape of the calibration or registration phantomin relationship to the patient, the operator and/or the OHMD.

Any other techniques known in the art, including as described in thisspecification, that can be used to determine the location, position,orientation, alignment, surfaces or shape of the calibration orregistration phantom in relationship to the patient, the operator and/orthe OHMD, can be used, including, but not limited to surgical navigationincluding optical or RF tracking, laser based distance measurements andthe like.

The calibration or registration phantom can be used for primary orback-up registration. Optionally, synchronized registration can be used,wherein, for example, more than one technique of registration is usedsimultaneously to maintain registration between virtual patient data andlive patient data, for example by simultaneously maintainingregistration between virtual patient data and live patient data usingone or more calibration or registration phantoms in conjunction withmaintaining registration using corresponding anatomic landmarks orsurfaces between virtual patient data and live patient data. Ifsynchronized registration is used, optionally, rules can be applied toresolve potential conflicts between a first and a second registrationtechnique for registering virtual and live patient data.

For example, with an image and/or video capture system integrated intoor attached to the OHMD or coupled to the OHMD, any change in theposition, location or orientation of the surgeon's or operator's head orbody will result in a change in the perspective view and visualized sizeand/or shape of the calibration or registration phantom. The change inperspective view and visualized size and/or shape of the calibration orregistration phantom can be measured and can be used to determine thechange in position, location or orientation of the surgeon's oroperator's head or body, which can then be used to maintain registrationbetween the virtual patient data and the live patient data, by movingthe virtual patient data into a position, location, orientation and/oralignment that ensures that even with the new position location ororientation of the surgeon's or operator's head or body the registrationis maintained and the virtual and the live patient data are, forexample, substantially superimposed or matched where desired. Similarly,when more than one OHMD is used, e.g. one for the primary surgeon, asecond OHMD for an assistant, a third OHMD for a resident, a fourth OHMDfor a scrub nurse and a fifth OHMD for a visitor, with an image and/orvideo capture system integrated into or attached to each of thedifferent OHMDs or coupled to each of the different OHMDs, any change inthe position, location or orientation of the user's or viewer's head orbody will result in a change in the perspective view and visualized sizeand/or shape of the calibration or registration phantom. The change inperspective view and visualized size and/or shape of the calibration orregistration phantom can be measured and can be used to determine thechange in position, location or orientation of the user's or viewer'shead or body, which can then be used to maintain registration betweenthe virtual patient data and the live patient data, by moving thevirtual patient data into a position, location, orientation and/oralignment that ensures that even with the new position location ororientation of the user's or viewer's head or body the registration ismaintained and the virtual and the live patient data are, for example,substantially superimposed or aligned or matched where desired, withsubstantially identical view angle of the virtual data of the patientseen by the viewer's left eye through the display of the OHMD unit andthe live data of the patient seen by the viewer's left eye through theOHMD unit and substantially identical view angle of the virtual data ofthe patient seen by the viewer's right eye through the display of theOHMD unit and the live data of the patient seen by the viewer's righteye through the OHMD unit for each of the OHMDs used. In someembodiments of the invention, the calibration or registration phantomcan be used to check the accuracy of an integrated or attached orcoupled or separate image and/or video capture system.

In a further embodiment of the invention, the calibration orregistration phantom can be used to calibrate an integrated or attachedor coupled or separate image and/or video capture system.

In some embodiments, the calibration or registration phantom can be usedto calibrate the IMU, e.g. for distance measurements, movement, distanceto object, since calibration or registration phantom includes knowngeometries, e.g. known distances or angles.

Registration of Virtual Patient Data and Live Patient Data Accountingfor Tissue Deformation

In some embodiments of the invention, tissue deformation, a shape changeor removal of tissue caused by the surgery or surgical instruments canbe simulated in the virtual data. The resultant simulated virtual datacan then be registered related to the live patient data, either beforeand/or after deformation, alteration of shape or removal of tissue ofthe live patient. The tissue deformation, shape change or removal oftissue caused by the surgery or surgical instruments can include theshape alteration or removal of one or more osteophytes or bone spurs orother bony anatomy or deformity. The virtual data of the patient and thelive data of the patient can be registered in a common coordinatesystem, for example with one or more OHMDs. Virtual and physicalsurgical instruments and implant components can also be registered inthe common coordinate system.

In some embodiments of the invention, the registration of virtualpatient data and live patient data using the techniques described hereincan be repeated after one or more surgical steps have been performed. Inthis case, the surgically altered tissue or tissue surface or tissuecontour or tissue perimeter or tissue volume or other tissue features inthe live patient can be matched to, superimposed onto and/or registeredwith the surgically altered tissue or tissue surface or tissue contouror tissue perimeter or tissue volume or other tissue features in thevirtual data of the patient, e.g. in a virtual surgical plan developedfor the patient. The matching, superimposing and/or registering of thelive data of the patient and the virtual data of the patient after thesurgical tissue alteration can be performed using the same techniquesdescribed in the foregoing or any of the other registration techniquesdescribed in the specification or any other registration technique knownin the art. Re-registration of live patient data and virtual patientdata can be particularly helpful if the surgical alteration or surgicalstep has led to some tissue deformation. For example, there-registration can be performed by matching, superimposing, and/orregistering tissues that have not been performed by the surgical step orsurgical alteration. Alternatively, the re-registration can be performedby matching, superimposing and/or registering deformed live patientdata, e.g. from surgically deformed tissue, with virtual patient datathat simulate the same tissue deformation after the virtual surgicalstep, e.g. an osteophyte or tissue removal.

Registration of Virtual Patient Data and Live Patient Data at MultipleTime Points, for Example at Different Stages of a Surgical Procedure

In some embodiments of the invention, registration of virtual patientdata and live patient data can occur at multiple time points, forexample during different phases of tissue removal or implantation of amedical device. For select or each time point, e.g. for select or allstages of the surgical procedure, the live data of the patient and thevirtual data of the patient can be registered in a common coordinatesystem, for example with one or more OHMDs. Virtual and physicalsurgical instruments can also be registered in the common coordinatesystem.

In knee replacement surgery or hip replacement surgery, for example,registration of virtual patient data and live patient data can beperformed using, for example, the femoral or tibial or acetabularsurface shape or using femoral or tibial or acetabular landmarks priorto the resection of any tissue. Optionally pins or other rigid fixationmarkers can be placed, for example in an area that will not besurgically resected during at least part of the surgical procedure. Theregistration of virtual and live patient data can be repeated usingdifferent registration sites, surfaces or landmarks after tissue hasbeen removed, e.g. after a burring of the articular surface has occurredor after a bone cut has been performed or after reaming has beenperformed or after one or more osteophytes or bone spurs or other bonyanatomy or deformity have been removed. The registration can now occurto a newly created landmark, created by the surgical procedure, or, forexample, a newly created surface, e.g. created by the surgicalprocedure. Such a newly created surface can be, for example, a planarsurface on the residual femur or tibia created by a bone cut. Optionallyimplanted pins or rigid fixation markers can be used to aid with theregistration of the virtual data after surgical alteration and the livedata of the patient altered by the surgery. Thus, the current inventionallows for multiple time point registration of virtual patient data andlive patient data, for example by registered virtual patient data to thelive patient data prior to surgical alteration and after one or moresurgical alterations. In this manner, it is possible to re-registermultiple times as surgical field changes.

The registration of virtual patient data and live patient data using thetechniques described herein can be repeated after one or more surgicalsteps have been performed. In this case, the surgically altered tissueor tissue surface or tissue contour or tissue perimeter or tissue volumeor other tissue features in the live patient can be matched to,superimposed onto and/or registered with the surgically altered tissueor tissue surface or tissue contour or tissue perimeter or tissue volumeor other tissue features in the virtual data of the patient, e.g. in avirtual surgical plan developed for the patient. The matching,superimposing and/or registering of the live data of the patient and thevirtual data of the patient after the surgical tissue alteration can beperformed using the same techniques described in the foregoing or any ofthe other registration techniques described in the specification or anyother registration technique known in the art.

Registration of Virtual Patient Data and Live Patient Data Using CADFiles or Data or 3D Files or Data, e.g. of a Medical Device

In some embodiments of the invention, a CAD file or CAD data of amedical device can be displayed by the OHMD and superimposed on livedata of the patient. The CAD file or CAD data can be a medical deviceintended for use or implantation during the surgical procedure. Any typeof CAD file or CAD data or any type of 3D file or 3D data of a medicaldevice, a surgical instrument or an implantable device can besuperimposed and registered in relationship to the live data of thepatient including normal anatomy or pathologic tissue, e.g. one or moreosteophytes or bone spurs or other bony anatomy or deformity orsoft-tissue or neoplastic tissue or abnormality in a common coordinatesystem, for example with one or more OHMDs.

Physical surgical instruments and implant components can also beregistered in the common coordinate system.

Medical devices can include non-biologic as well as biologic devices,e.g. tissue scaffolds, cells, cell matrices etc. that can be implantedin a human body.

In some embodiments of the invention, multiple CAD files and/or 3D filesof virtual data can be superimposed onto the live data of the patient.For example, CAD files can be CAD files of a medical device available indifferent sizes or shapes. Virtual 2D or 3D data of the patient, forexample obtained from a preoperative imaging test, can be superimposedonto live data of the patient, e.g. a surgical site. The surgeon canthen optionally introduce a 3D CAD file of a medical device into thedisplay by the OHMD. The surgeon can check the size or shape of themedical device in relationship to the virtual 2D or 3D data of thepatient and/or the live data of the patient. If the surgeon is notsatisfied with the projected size or shape of the medical device inrelationship to the virtual 2D or 3D data of the patient and/or the livedata of the patient, the surgeon can select a different CAD file of amedical device with a different size and/or shape, project the CAD fileoptionally onto the virtual 2D or 3D data of the patient and the livedata of the patient in the OHMD display and repeat the process as manytimes as needed until the surgeon is satisfied with the resultant sizeor shape of the selected medical device in relationship to the virtual2D or 3D data of the patient and/or the live data of the patient.

The registration of virtual patient data and live patient data using thetechniques described herein can be repeated after one or more surgicalsteps have been performed. In this case, the surgically altered tissueor tissue surface or tissue contour or tissue perimeter or tissue volumeor other tissue features in the live patient can be matched to,superimposed onto and/or registered with the surgically altered tissueor tissue surface or tissue contour or tissue perimeter or tissue volumeor other tissue features in the virtual data of the patient, e.g. in avirtual surgical plan developed for the patient. The matching,superimposing and/or registering of the live data of the patient and thevirtual data of the patient after the surgical tissue alteration can beperformed using the same techniques described in the foregoing or any ofthe other registration techniques described in the specification or anyother registration technique known in the art. For example, CAD filessimulating the virtual surgical step or surgical alteration in thevirtual patient data can be matched, superimposed or registered withlive patient data after the physical surgical step or surgicalalteration in the live patient. In this manner, live and virtual datacan be re-registered after the surgical step or surgical alteration.

Registration of Virtual Patient Data and Live Patient Data UsingNon-Anatomic Data

Registration of virtual data of the patient and live data of the patientcan be performed using data other than anatomic or pathologicstructures. Registration can be performed, for example, based on motiondata, kinematic data (for example to determine the center of rotation ofa joint in the live data which can then be registered to an estimate orsimulated center of rotation in the virtual data of the patient).Registration can be performed using metabolic data, for example using anarea of high 18 FDG-PET uptake in a PET scan or PET-MRI or PET CT, whichcan be, for example matched to an area of increased body temperature ina target surgical site. Registration can be performed using functionaldata, e.g. using functional MRI studies. Virtual data and live data ofthe patient can be registered in a common coordinate system, for examplewith one or more OHMDs. Virtual and physical surgical instruments andimplant components can also be registered in the common coordinatesystem.

Optionally, different types of data, e.g. anatomic, motion, kinematic,metabolic, functional, temperature and/or vascular flow data can be usedalone or in combination for registered virtual and live data of thepatient.

The registration of virtual patient data and live patient data using thetechniques described herein can be repeated after one or more surgicalsteps have been performed using non-anatomic data. In this case, thesurgically altered tissue or tissue surface or tissue contour or tissueperimeter or tissue volume or other tissue features in the live patientcan be matched to, superimposed onto and/or registered with thesurgically altered tissue or tissue surface or tissue contour or tissueperimeter or tissue volume or other tissue features in the virtual dataof the patient, e.g. in a virtual surgical plan developed for thepatient, optionally using non-anatomic data. The matching, superimposingand/or registering of the live data of the patient and the virtual dataof the patient after the surgical tissue alteration can be performedusing the same techniques described in the foregoing or any of the otherregistration techniques described in the specification or any otherregistration technique known in the art.

Registration of Virtual Patient Data and Live Patient Data afterPerforming One or More Surgical Alterations to the Tissue or theSurgical Site

In some embodiments of the invention, the registration of virtualpatient data and live patient data using the techniques described hereincan be repeated after one or more surgical steps have been performed andvirtual data and live data of the patient can be registered in a commoncoordinate system after select steps or each surgical step or tissuealteration, for example with one or more OHMDs. Virtual and physicalsurgical instruments and implant components can also be registered inthe common coordinate system after select steps or each surgical step ortissue alteration. The surgically altered tissue or tissue surface ortissue contour or tissue perimeter or tissue volume or other tissuefeatures in the live patient can be matched to, superimposed onto and/orregistered with the surgically altered tissue or tissue surface ortissue contour or tissue perimeter or tissue volume or other tissuefeatures in the virtual data of the patient, e.g. in a virtual surgicalplan developed for the patient. The matching, superimposing and/orregistering of the live data of the patient and the virtual data of thepatient after the surgical tissue alteration can be performed using thesame techniques described in the foregoing or any of the otherregistration techniques described in the specification or any otherregistration technique known in the art.

The matching, superimposing and/or registering of the live data of thepatient and the virtual data of the patient after the surgical tissuealteration can be manual, semi-automatic or automatic using informationabout the surgically altered tissue or tissue surface or tissue contouror tissue perimeter or tissue volume or other tissue features. Automatedre-registration can, for example, be performed using an image and/orvideo capture system integrated into, attached to or separate from theOHMD which can capture information about the surgically altered tissueor tissue surface or tissue contour or tissue perimeter or tissue volumeor other tissue features in the live patient data after the surgicalalteration and compare the information to information in the virtualdata of the patient, e.g. for the virtual data after performing thecomparable step in a virtual surgical plan.

The surgical alteration or surgical steps can include, but are notlimited to the procedures in Table 6:

-   -   TABLE 6: Exemplary surgical alterations or steps applied to        various patient tissues, e.g. bone, cartilage, ligaments,        tendons, joint capsule, skin, fat, organ tissue, e.g. liver,        spleen, kidney, intestines, gallbladder, lung, heart, thyroid,        brain etc.    -   Cutting, e.g. a bone cut    -   Sawing, e.g. sawing a bone with a saw    -   Milling, e.g. milling a bone with a mill    -   Reaming, e.g. reaming a bone with a reamer    -   Impacting, e.g. impacting a bone with an impactor    -   Drilling, e.g. drilling a bone with a drill    -   Pinning, e.g. pinning a bone with a pin    -   Radiofrequency ablation    -   Heat ablation    -   Cryoablation    -   Cauterization    -   Tissue resection    -   Tissue removal    -   Resection of a neoplasm    -   Fracture fixation    -   Trauma repair    -   Trauma reconstruction    -   Soft-tissue repair    -   Soft-tissue reconstruction    -   Tissue grafting    -   Placement of a registration marker or calibration phantom on the        tissue surface or inside the tissue    -   Placement of a surgical instrument, e.g. a pin or a saw    -   Placement of a medical implant or a component thereof, e.g. a        biopsy needle, pedicle needle, pedicle screw, a spinal rod, a        component of a knee replacement system, a component of a hip        replacement system, a component of a shoulder replacement        system, a component of an ankle replacement system    -   Placement/injection of bone cement or other substances,        hardening or non hardening    -   Placement of a trial implant    -   Placement of a tissue graft    -   Placement of a tissue matrix    -   Placement of a transplant    -   Placement of a catheter, e.g. an indwelling catheter    -   Placement or injection of cells, e.g. stem cells    -   Injection of a drug

Optionally, the registration procedures described herein can be repeatedafter performing a surgical step. Optionally, the registrationprocedures described herein can be repeated after multiple surgicalsteps. Optionally, the registration procedures described herein can berepeated after each surgical step. Optionally, the registrationprocedures described herein can be repeated after major surgical steps.Optionally, the registration procedures described herein can be repeatedwhen the surgeon wants to achieve high surgical accuracy. Optionally,the registration procedures described herein can be performed orrepeated when the surgeon is concerned that the initial registrationperformed prior to the surgical step or surgical alteration was notaccurate or is not accurate any longer or is affected by the surgicalstep or surgical alteration.

In some embodiments of the invention, the change on the patient's tissueinduced by the surgical alteration or the surgical step can be known orestimated, for example as part of the virtual surgical plan using thevirtual data of the patient. Surgical alterations and/or surgical stepsapplied to patient tissues can include any of the surgical alterationsand/or surgical steps listed in the examples in Table 6, although anyalteration to a patient's tissue known in the art can be included. Thealteration and/or the change induced on the patient's tissue by thesurgical alteration or surgical step can be estimated, for example inthe virtual surgical plan and/or the virtual data of the patient.Exemplary changes induced on the patient's tissue by the surgicalalteration or surgical step are tabulated in Table 7, which is onlyexemplary in nature and in no way meant to be limiting of the invention:

TABLE 7: Exemplary changes induced on the patient's tissue by a surgicalalteration or surgical step. These changes can be induced in the livepatient. These changes can also be planned/intended or simulated, e.g.for projection by one or more OHMDs, e.g. in a virtual surgical plan.

-   -   Change in tissue surface area    -   Change in tissue volume    -   Change in tissue surface shape    -   Change in tissue surface topography    -   Change in tissue perimeter (e.g. from uncut to cut surface, or        from cut surface 1 to cut surface 2)    -   Change in tissue surface roughness    -   Change in tissue surface texture    -   Change in tissue surface color    -   Change in tissue surface reflexivity (e.g. reflected light or        ultrasound)    -   Change in tissue surface area with different color (e.g. color        change induced by surgical alteration)    -   Change in tissue surface perimeter, e.g. cut vs. uncut tissue        surface    -   Change in tissue temperature    -   Change in tissue elasticity    -   Change in tissue composition, e.g. fat content (e.g. marrow fat        on a cut bone surface)

Any of the foregoing changes can include all of the tissue or only aportion of the tissue. The embodiments of the invention can be directedtowards all of the tissue or only partial tissue or portions of thetissue.

Following initial registration of the live data of the patient with thevirtual data of the patient using any of the techniques described in thespecification or known in the art, a first or any subsequent surgicalalteration or surgical step can be performed inducing changes to thepatient's tissue. The surgical alteration or surgical step can beperformed with optional guidance through the OHMD display, e.g. bydisplaying one or more of virtual surgical tool, virtual surgicalinstrument including a virtual surgical guide or cut block, virtualtrial implant, virtual implant component, virtual implant or virtualdevice, all optionally selected from a virtual library, a predeterminedstart point, predetermined start position, predetermined startorientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration.

Once a surgical alteration or surgical step has been performed orinduced on a patient's tissue in the live patient, the physical changesinduced or the resultant tissue appearance and/or tissueproperties/characteristics can be determined in the live data of thepatient/the live patient. The physical changes induced or the resultanttissue appearance and/or tissue properties/characteristics can bedetermined in the live data of the patient/the live patient using anytechnique known in the art for assessing tissue appearance, tissueproperties and/or characteristics including, for example, area, volume,shape, topography, roughness, texture, color, reflexivity, area withdifferent color, perimeter, temperature, elasticity, and/or composition.For example, an image and/or video capture system integrated into,attached to or separate from an OHMD can be used to assess one or moreof an area, shape, topography, roughness, texture, color, reflexivity,area with different color, perimeter, temperature, elasticity, and/orcomposition of a surgically altered tissue. Tissue probes, e.g.temperature probes, elasticity probes, can be used to assesscharacteristics and/or properties of the surgically altered tissue.Mechanical probes, e.g. with one or more attached optical markers, LEDs,infrared markers, retroreflective markers, RF markers, navigationmarkers and/or IMUs can be used to touch the tissue surface or perimeterand, for example, to circle a perimeter or to follow and assess a tissuetopography of a surgically altered tissue.

The physical appearance, properties and/or characteristics of thesurgically altered tissue can be assessed using any of the foregoingtechniques or any of the techniques described in the specification orknown in the art. The physical appearance, properties and/orcharacteristics of the surgically altered tissue can optionally becompared to the estimated or intended change or post-alterationappearance, e.g. surface area, volume, shape, topography, propertiesand/or characteristics of the tissue in the virtual data of the patient,for example the virtual surgical plan. If there are differences betweenthe physical change in the physical surgically altered tissue and thevirtually intended change in the virtually surgically altered tissue orif there are differences in the appearance, properties and/orcharacteristics of the physical surgically altered tissue and thevirtually altered tissue, e.g. in the virtual data of the patient and/orthe virtual surgical plan, the magnitude of the differences can beassessed: If the differences are deemed to be insignificant, forexample, if they fall below an, optionally predefined, threshold indistance or angular deviation, the surgical procedure and subsequentsurgical steps can continue as originally planned, e.g. in the virtualsurgical plan. If the differences are deemed to be significant, forexample, if they fall above an, optionally predefined, threshold indistance or angular deviation, the surgeon or the operator can haveseveral options. The process and the options are also shown inillustrative form in FIG. 6: The surgeon can perform a surgical step 80.The surgeon can then assess the actual changes induced in the livepatient 81. The surgeon can compare the actual changes induced in thelive patient with the predetermined changes in the virtual data of thepatient, e.g. in a virtual surgical plan or in a virtual 3D display 82.The magnitude of the difference(s) between the actual and thepredetermined changes can be determined 83. If they are acceptable 84,the surgeon can perform the next surgical step 85. Optionally 85, thesteps 81, 82, 83 can be repeated for the next surgical step. If thedifference(s) between the actual and the predetermined changes are notacceptable 86, the surgeon has several means of addressing thedifference(s), modify the last surgical step 87, modify the nextsurgical step 88, modify the virtual surgical plan 89, modify theregistration of the virtual data of the patient in relationship to thelive data of the patient 90, or apply registration correction 91. Afterthe last surgical step has been modified 87, optionally 92, the steps81, 82, 83 can be repeated for the next surgical step.

A). Modify the Last Surgical Step so that the physical appearance,physical properties and/or physical characteristics (including, forexample, shape and dimensions, cut plane, perimeter of a cutplane/tissue plane, drill depth, angle, rotation, implant site etc.) ofthe surgically altered tissue in the live patient after the modificationis more similar to and, optionally, more closely replicates the intendedvirtual appearance, virtual properties and/or virtual characteristics inthe virtual data of the patient, e.g. a virtual surgical plan of thepatient. This option can, for example, be chosen if the operator orsurgeon is of the opinion that the last surgical step was subject to aninaccuracy, e.g. by a fluttering or deviating saw blade or a misalignedpin or a misaligned reamer or impactor or other problem, and shouldcorrect the inaccuracy. Once the modification has been completed, thesurgeon or operator can again assess the physical change, physicalappearance, physical properties and/or physical characteristics of thesurgically altered tissue and compared it to the estimated or intendedvirtual change, virtual appearance, virtual properties and/or virtualcharacteristics of the tissue in the virtual data of the patient, forexample the virtual surgical plan. Depending on the result of theassessment, the surgeon or operator can optionally repeat option A, orrevert to options B or C.

B). Modify the Next Surgical Step(s) so that the physical appearance,physical properties and/or physical characteristics (including, forexample, shape and dimensions, cut plane, perimeter of a cutplane/tissue plane, drill depth, angle, rotation, implant site etc.) ofthe surgically altered tissue in the live patient after the modificationin the next surgical step(s) is more similar to and, optionally, moreclosely replicates the intended virtual appearance, virtual propertiesand/or virtual characteristics in the virtual data of the patient, e.g.a virtual surgical plan of the patient after the virtual modification inthe next virtual surgical step(s). This option can, for example, bechosen if the operator or surgeon is of the opinion that the lastsurgical step was subject to an inaccuracy, e.g. by a fluttering ordeviating saw blade or a misaligned pin or a misaligned reamer orimpactor or other problem, and he or she should correct the inaccuracyin the next surgical step(s). Once the modification has been completedwith the next surgical step(s), the surgeon or operator can again assessthe physical change, physical appearance, physical properties and/orphysical characteristics of the surgically altered tissue and comparedit to the estimated or intended virtual change, virtual appearance,virtual properties and/or virtual characteristics of the tissue in thevirtual data of the patient, for example the virtual surgical plan.Depending on the result of the assessment, the surgeon or operator canoptionally repeat option A and/or B and/or revert to options C and/or Dand/or E.

C). Modify the Virtual Surgical Plan of the patient so that the virtualappearance, virtual properties and/or virtual characteristics(including, for example, shape, volume and dimensions, cut plane,perimeter or surface/surface area of a cut plane/tissue plane, drilldepth, angle, rotation, implant site etc.) of the surgically alteredtissue in the virtual data of the patient after the modification is/aremore similar to and, optionally, more closely replicates the physicalappearance, physical properties and/or physical characteristics in thephysical live data of the patient after the physical surgicalalteration. This option can, for example, be chosen if the operator orsurgeon is of the opinion that the last surgical step was accurate oraccounted for unexpected variations in tissue conditions that were notaccounted for in the virtual surgical plan. Such unexpected variationsin tissue conditions can, for example, be ligament laxity or tightnessas can be observed, for example, in knee replacement surgery or hipreplacement or other joint replacement surgeries. If the modifiedsurgical plan is modified in this manner, all subsequent virtualsurgical steps can then be referenced off the last or preceding physicalsurgical step, thereby maintaining continuity of the procedure. The OHMDcan then be used for projecting all or some of the subsequent virtualsurgical steps, e.g. by projecting one or more of virtual surgical tool,virtual surgical instrument, virtual trial implant, virtual implantcomponent, virtual implant or virtual device, all optionally selectedfrom a virtual library, a predetermined start point, predetermined startposition, predetermined start orientation or alignment, predeterminedintermediate point(s), predetermined intermediate position(s),predetermined intermediate orientation or alignment, predetermined endpoint, predetermined end position, predetermined end orientation oralignment, predetermined path, predetermined plane, predetermined cutplane, predetermined contour or outline or cross-section or surfacefeatures or shape or projection, predetermined depth marker or depthgauge, predetermined angle or orientation or rotation marker,predetermined axis, e.g. rotation axis, flexion axis, extension axis,predetermined axis of the virtual surgical tool, virtual surgicalinstrument including virtual surgical guide or cut block, virtual trialimplant, virtual implant component, implant or device, non-visualizedportions for one or more devices or implants or implant components orsurgical instruments or surgical tools, and/or one or more of apredetermined tissue change or alteration. The subsequent virtualsurgical steps are thus modified to allow completion of the procedureand, optionally, placement of an implant or implant component or deviceor graft or transplant taking into account the one or more modifiedpreceding physical surgical steps. Optionally, the modified subsequentvirtual surgical steps can be further modified based on local tissueconditions/characteristics after the virtual or physical modification,for example, if subsequent surgical steps were to fall into a tissuevoid or would result in impairment of implant component placement.

D). Modify the Registration of the Virtual Data of the Patient inRelationship to the Live Data of the Patient. The operator or surgeoncan optionally repeat the registration procedure using any of thetechniques described in the specification or known in the art forregistering the virtual data of the patient, including, for example thevirtual surgical plan, in relationship to the live data of the patientafter the physical surgical alteration. Once the virtual data of thepatient and the live data of the patient after the surgical alterationhave been re-registered, all subsequent virtual surgical steps displayedby the OHMD and any related virtual surgical plan can be referenced offthe re-registration of the virtual and live data of the patient. Forexample, the OHMD can then be used after the re-registration forprojecting all subsequent virtual surgical steps, e.g. by projecting oneor more of virtual surgical tool, virtual surgical instrument, virtualtrial implant, virtual implant component, virtual implant or virtualdevice, all optionally selected from a virtual library, a predeterminedstart point, predetermined start position, predetermined startorientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration.

E). Apply Registration Correction. If there are differences between thephysical change in the physical surgically altered tissue and thevirtually intended change in the virtually surgically altered tissue orif there are differences in the appearance, properties and/orcharacteristics of the physical surgically altered tissue and thevirtually altered tissue, e.g. in the virtual data of the patient and/orthe virtual surgical plan, the magnitude of the differences can beassessed and can be used to apply a coordinate correction, coordinateadjustment or coordinate transfer of registration of the virtual data ofthe patient, including, optionally, the virtual surgical plan, and thelive data of the patient, e.g. for any subsequent surgical steps orsurgical procedures. For example, the OHMD can then project/display allsubsequent virtual surgical steps using the coordinate correction oradjustment or transfer, e.g. by projecting one or more of virtualsurgical tool, virtual surgical instrument including virtual surgicalguide or cut block, virtual trial implant, virtual implant component,virtual implant or virtual device, all optionally selected from avirtual library, a predetermined start point, predetermined startposition, predetermined start orientation or alignment, predeterminedintermediate point(s), predetermined intermediate position(s),predetermined intermediate orientation or alignment, predetermined endpoint, predetermined end position, predetermined end orientation oralignment, predetermined path, predetermined plane, predetermined cutplane, predetermined contour or outline or cross-section or surfacefeatures or shape or projection, predetermined depth marker or depthgauge, predetermined angle or orientation or rotation marker,predetermined axis, e.g. rotation axis, flexion axis, extension axis,predetermined axis of the virtual surgical tool, virtual surgicalinstrument including virtual surgical guide or cut block, virtual trialimplant, virtual implant component, implant or device, non-visualizedportions for one or more devices or implants or implant components orsurgical instruments or surgical tools, and/or one or more of apredetermined tissue change or alteration using the coordinatecorrection, adjustment and/or transfer.

Any combinations of the foregoing Options A, B, C, D and/or E arepossible.

If an image and/or video capture system is used to measure/capture thephysical changes, e.g. change in surface/surface area, perimeter,perimeter shape, and/or shape of the cut surface or otherwise modifiedor altered surface, the data/images captured by the image and/or videocapture system can be corrected for any angular distortion orprojection, for example if the camera(s) is/are positioned at an angleother than 90 degrees relative to the cut surface or otherwise modifiedor altered surface. Similarly, the physical changes measured by theimage and/or video capture system, e.g. the size of the surface/surfacearea, perimeter, perimeter shape, and/or shape of the cut surface orotherwise modified or altered surface, can be corrected or adjusted forthe distance between the camera or image and/or video capture system andthe changed surface/surface area, perimeter, perimeter shape, and/orshape of the cut surface or otherwise modified or altered surface. Theangle and/or the distance of the image and/or video capture system tothe physical changes, e.g. surface/surface area, perimeter, perimetershape, and/or shape of the cut surface or otherwise modified or alteredsurface, can be assessed, for example, using one or more RF markers,optical markers, navigation markers including, but not limited to,infrared markers, retroreflective markers, RF markers, LEDs, and/or IMUsattached to the image and/or video capture system, and/or the OHMD,and/or the patient, and/or the cut, modified or altered surface.

For example, in a knee replacement, hip replacement or shoulderreplacement procedure, a bone cut can be applied, optionally usingvirtual guidance of a bone saw by the OHMD, to a distal femur, proximaltibia, proximal femur or proximal humerus. The position, alignmentand/or orientation of the bone cut, including, optionally, thesurface/surface area, perimeter, perimeter shape, and/or shape of thecut surface can then be assessed in the live patient, for example usingan image and/or video capture system integrated into, attached to orseparate from the OHMD or using one or more probes, optionally with oneor more attached optical markers, navigation markers including, but notlimited to, infrared markers, retroreflective markers, RF markers, LEDs,or IMUs.

If the physical position, alignment, orientation, surface, surface area,perimeter, perimeter shape, and/or shape of the cut surface differ fromthe virtually intended/projected position, alignment, orientation,surface, surface area, perimeter, perimeter shape, and/or shape of thecut surface, the software can, optionally, determine a virtuallymodified position, alignment, orientation, surface, surface area,perimeter, perimeter shape, and/or shape of the cut surface that wouldmore closely resemble the physical position, alignment, orientation,surface, surface area, perimeter, perimeter shape, and/or shape of thecut surface. The difference in coordinates between the virtuallymodified position, alignment, orientation, surface, surface area,perimeter, perimeter shape, and/or shape of the cut surface and thephysical position, alignment, orientation, surface, surface area,perimeter, perimeter shape, and/or shape of the cut surface can then beused to determine any coordinate correction, adjustment or transfer forsubsequent virtual surgical steps. The coordinate correction, adjustmentor transfer can then by applied to the OHMD displays, for example whenthe OHMD displays in any subsequent surgical steps one or more ofvirtual surgical tool, virtual surgical instrument, virtual trialimplant, virtual implant component, virtual implant or virtual device,all optionally selected from a virtual library, a predetermined startpoint, predetermined start position, predetermined start orientation oralignment, predetermined intermediate point(s), predeterminedintermediate position(s), predetermined intermediate orientation oralignment, predetermined end point, predetermined end position,predetermined end orientation or alignment, predetermined path,predetermined plane, predetermined cut plane, predetermined contour oroutline or cross-section or surface features or shape or projection,predetermined depth marker or depth gauge, predetermined angle ororientation or rotation marker, predetermined axis, e.g. rotation axis,flexion axis, extension axis, predetermined axis of the virtual surgicaltool, virtual surgical instrument including virtual surgical guide orcut block, virtual trial implant, virtual implant component, implant ordevice, non-visualized portions for one or more devices or implants orimplant components or surgical instruments or surgical tools, and/or oneor more of a predetermined tissue change or alteration using thecoordinate correction, adjustment and/or transfer.

The following is an exemplary description of a portion of a hipreplacement procedure shown in the illustrative example in FIG. 7A-H,where the surgeon elects to make a correction to the proximal femoralcut prior to proceeding with the subsequent steps of the procedure. Thisexample is in no way meant to be limiting, but only illustrative ofcertain aspects of the invention.

FIG. 7A shows a view of a predetermined femoral neck 95 cut or a virtualsurgical plan, as optionally displayed by an OHMD in 2D or 3D,stereoscopic or non-stereoscopic, including using a digital holographicrepresentation with a system such as a Microsoft Hololens (Microsoft,Redmond, Wash.). The OHMD can display a virtual predetermined path orplane (broken line) 96 for a saw blade selected to make the proximalfemoral cut in this example. The OHMD can also display a digitalhologram of the virtual femoral neck cut. The virtual projected path fora physical saw blade to make the proximal femoral neck cut and thevirtual femoral neck cut can be the same; they can also be different,for example accounting for the thickness of the saw blade. For example,if a saw blade thickness is 2.0 mm, the predetermined path can be moved,e.g. in a proximal femur for hip replacement proximally, 1.00 mm or moreto account for bone lost from the sawing so that the virtual femoralbone cut accounts for the bone lost by the sawing.

The display of the predetermined path can be in 2D or in 3D,stereoscopic or non-stereoscopic. The surgeon can align the physical sawblade with the predetermined path and the surgeon can then advance thesaw blade while keeping the saw blade substantially aligned with thepredetermined path as shown by the OHMD. Rather than display thepredetermined path, the OHMD can also display a virtual bone saw alignedto make the virtual bone cut (optionally accounting for bone lost fromthe cutting or sawing) and the surgeon can align the physical bone sawwith the virtual bone saw and make the cut.

FIG. 7B shows a cross-section or top view of the intended virtualfemoral neck cut (broken outline) 97, for example as developed in thevirtual surgical plan. The perimeter and/or cross-section and/or surfacearea and/or shape of the virtually cut femur, for example simulatedusing data from a pre-operative imaging study of the patient, e.g. CT orMRI, is relatively round in this example with slightly greater diameterin medial-lateral direction.

FIG. 7C shows the physical femoral neck cut 98 made in the live patient(straight solid line). The physical femoral neck cut is not aligned withthe virtually projected or intended path for the saw blade and it is notaligned with the virtual femoral neck cut, for example in the virtualsurgical plan, in this example. This can happen for various reasons inlive surgery, for example unexpectedly sclerotic areas of bone thatcause saw blade deviation. The difference in alignment between thevirtually intended bone cut and the physical femoral bone cut can bedifficult to detect for the surgeon intraoperatively, for example if thesurgical field is small and deep seated, obscured or hidden or haslimited lighting or if only a small portion of the cut bone is exposed.

FIG. 7D shows the top view or cross-section of the physical femoral neckcut (solid outline) 99. The perimeter and/or cross-section and/orsurface area and/or shape of the physical femoral neck cut is differentthan the perimeter and/or cross-section and/or surface area and/or shapeof the virtually planned cut femur. It is more elliptical or oblong inmedial-lateral direction. The perimeter and/or cross-section and/orsurface area and/or shape of the physical cut proximal femur can bedetected, for example using an image and/or video capture systemintegrated into, attached to or separate from the OHMD or using amechanical or optical probe, for example using RF, optical, navigationand other markers. It can then be compared to the perimeter and/orcross-section and/or surface area and/or shape of the virtual cutsurface.

In FIG. 7E, once the perimeter and/or cross-section and/or surface areaand/or shape of the physical cut proximal femur has been detected, forexample using an image and/or video capture system integrated into,attached to or separate from the OHMD or using a mechanical or opticalprobe, for example using RF, optical, navigation and other markers, acorresponding perimeter and/or cross-section and/or surface area and/orshape of the physical cut proximal femur can be identified in thevirtual data of the patient (broken outline) 100, for example usingimage processing algorithms known in the art.

In FIG. 7F, once the corresponding perimeter and/or cross-section and/orsurface area and/or shape has been identified in the virtual data of thepatient (FIG. 7E), a new, substitute virtual femur cut 101 whichapproximates the intended femoral cut can be identified in the virtualdata of the patient. The difference in position, location, orientation,coronal, sagittal, axial angle/angulation between the originally plannedor predetermined virtual femoral bone cut and the substitute, newvirtual femoral bone cut can be determined. Depending on the severityand/or clinical significance of the difference between the originallyplanned or predetermined virtual femoral bone cut 96 and the substitute,new virtual femoral bone cut 101, corresponding to the physical femoralbone cut 98 executed in the patient, the surgeon can then decide orchose between or combine any of the preceding Options A-E, e.g. modifythe last surgical step, modify the next surgical step(s), modify thevirtual surgical plan of the patient, modify the registration of thevirtual data of the patient in relationship to the live data of thepatient, and/or apply registration correction or combinations thereof.

In FIG. 7G, the surgeon can elect to modify the last surgical step andcorrect the proximal femoral cut by applying a correction in thealignment and direction of the saw blade. The resultant correctedphysical proximal femoral bone cut 102 can then closely approximate theoriginally intended, virtually planned, projected proximal femoral bonecut 97.

FIG. 7H shows that the perimeter and/or cross-section and/or surfacearea and/or shape of the corrected physical proximal femoral bone cut103 approximates the perimeter and/or cross-section and/or surface areaand/or shape of the original virtually planned proximal femoral bonecut.

In the example of a knee replacement, it is not uncommon that the distalfemoral cut is not falling onto its intended location. For example,dense sclerotic bone underneath the arthritic area can cause a saw bladeto deflect, thereby changing the angulation of the distal femoral cut.Once the distal femoral cut has been completed, the perimeter and/orcross-section and/or surface area and/or shape of the physical cutdistal femoral bone can be assessed, for example using an image and/orvideo capture system integrated into, attached to or separate from theOHMD and/or using a laser scanner and/or 3D scanner and/or using one ormore probes which can, for example, touch and/or follow the cut femoralbone optionally with one or more attached optical markers, LEDs,navigation markers including, but not limited to, infrared markers,retroreflective markers, RF markers, and/or IMUs. The perimeter and/orcross-section and/or surface area and/or shape of the of the physicalcut distal femoral bone in the live patient can then be compared to theperimeter and/or cross-section and/or surface area and/or shape of theof the virtual cut distal femoral bone, for example in the virtualsurgical plan of the patient. The perimeter and/or cross-section and/orsurface area and/or shape of the physical cut distal femoral bone can beused to identify a corresponding perimeter and/or cross-section and/orsurface area and/or shape of the virtual distal femoral bone or acorresponding virtual cut plane in the virtual data of the patient thatcan yield a similar perimeter and/or cross-section and/or surface areaand/or shape of the virtual distal femoral bone.

If the difference between the physical cut distal femoral bone and thevirtual cut distal femoral bone is below a certain threshold, e.g. 1, 2,3 or more millimeter in cut depth from the distal femoral surface,and/or 1 degree, 2 degrees, 3 degrees or more in angulation, the surgerycan proceed as originally planned. If the difference between thephysical cut distal femoral bone and the virtual cut distal femoral boneis above a certain threshold, e.g. 1, 2, 3 or more millimeter in cutdepth from the distal femoral surface, and/or 1 degree, 2 degrees, 3degrees or more in angulation, the surgeon or operator can then decideor chose between the preceding Options A-E, e.g. modify the lastsurgical step, e.g. recut the distal femoral bone, optionally with useof thicker tibial inserts to compensate for the greater bone loss or areduced tibial cut depth, modify one of the next surgical steps, e.g.cut the tibia to account for greater or lesser femoral bone loss and/ordifferent femoral component angulation (e.g. in the sagittal plane or inthe coronal plane (e.g. with different femoral mechanical axis alignmentoptionally corrected on the tibial side with different tibial mechanicalaxis alignment), modify the virtual surgical plan of the patient, modifythe registration of the virtual data of the patient in relationship tothe live data of the patient, and/or apply registration correction orcombinations thereof.

FIG. 8A shows a predetermined distal femoral cut, for example as part ofa view of virtual surgical plan, as optionally displayed by OHMD in 2Dor 3D, non-stereoscopic or stereoscopic. The OHMD can display a virtualintended path or plane 110 for a physical saw blade selected to make thedistal femoral cut in this example. The virtual/projected path or planefor the physical saw blade to make the distal femoral cut and thevirtual distal femoral cut can coincide; they can also be different, forexample accounting for the thickness of the saw blade. For example, if asaw blade thickness is 2.0 mm, the predetermined path can be moved, e.g.in a distal femur for knee replacement proximally, 1.00 mm or more toaccount for bone lost from the sawing so that the virtual femoral bonecut accounts for the bone lost by the sawing.

The display of the predetermined path can be in 2D or in 3D,stereoscopic or non-stereoscopic. The surgeon can align the physical sawblade with the predetermined path and the surgeon can then advance thesaw blade while keeping the saw blade substantially aligned with thepredetermined path or plane as shown by the OHMD. Rather than displaythe predetermined path or plane, the OHMD can also display a virtualbone saw aligned to make the virtual bone cut (optionally accounting forbone lost from the cutting or sawing) and the surgeon can align thephysical bone saw with the virtual bone saw and make the cut.

FIG. 8B shows a cross-section or view of the intended virtual distalfemoral cut 111, for example as developed in the virtual surgical plan.The perimeter and/or cross-section and/or surface area and/or shape ofthe virtually cut femur, for example simulated using data from apre-operative imaging study of the patient, e.g. CT or MRI or ultrasoundor x-rays, is shown.

FIG. 8C shows the physical distal femoral cut made in the live patient112. The physical distal femoral cut is not aligned with the virtuallypredetermined path for the saw blade and it is not aligned with thevirtual distal femoral cut in the virtual surgical plan in this example.This can happen for various reasons in live surgery, for exampleunexpectedly sclerotic areas of bone that cause saw blade deviation. Thedifference in alignment between the virtually intended bone cut and thephysical femoral bone cut can be difficult to detect for the surgeonintraoperatively.

FIG. 8D shows the view or cross-section of the physical distal femoralcut 113. The perimeter and/or cross-section and/or surface area and/orshape of the physical distal femoral cut is different than the perimeterand/or cross-section and/or surface area and/or shape of the virtuallyplanned cut femur. The perimeter and/or cross-section and/or surfacearea and/or shape of the physical cut distal femoral cut can bedetected, for example using an image and/or video capture systemintegrated into, attached to or separate from the OHMD or using amechanical or optical probe, for example using RF, optical, navigationand other markers. It can then be compared to the perimeter and/orcross-section and/or surface area and/or shape of the virtual cutsurface.

FIG. 8E. Once the perimeter and/or cross-section and/or surface areaand/or shape of the physical cut distal femur has been detected, forexample using an image and/or video capture system integrated into,attached to or separate from the OHMD or using a laser scanner and/or 3Dscanner or using a mechanical or optical probe, for example using RF,optical, navigation and other markers, a corresponding perimeter and/orcross-section and/or surface area and/or shape of the physical cutdistal femur can be identified in the virtual data of the patient 114,for example using image processing algorithms known in the art.

FIG. 8F. Once the corresponding perimeter and/or cross-section and/orsurface area and/or shape has been identified in the virtual data of thepatient (FIG. 8E), a new, substitute virtual femur cut whichapproximates the physical femoral cut can be identified 115 in thevirtual data of the patient. The difference in position, location,orientation, coronal, sagittal, axial angle/angulation between theoriginally planned/predetermined virtual femoral bone cut and thesubstitute, new virtual femoral bone cut can be determined. Depending onthe severity and/or clinical significance of the difference between theoriginally planned or predetermined virtual femoral bone cut and thephysical femoral bone cut executed in the patient, the surgeon can thendecide or chose between or combine any of the preceding options A-E,e.g. modify the last surgical step, modify the next surgical step(s),modify the virtual surgical plan of the patient, modify the registrationof the virtual data of the patient in relationship to the live data ofthe patient, and/or apply registration correction or combinationsthereof.

FIG. 8G The surgeon can elect to modify the last surgical step andcorrect the distal femoral cut by applying a correction in the alignmentand direction of the saw blade, which can be, for example, in thesagittal plane (as shown in this example) or in the coronal plane if thephysical cut was misaligned in the coronal plane. The resultantcorrected physical distal femoral bone cut 116 can then closelyapproximate the originally intended, virtually planned, projected distalfemoral bone cut.

FIG. 8H shows that the perimeter and/or cross-section and/or surfacearea and/or shape of the corrected physical distal femoral bone cut 117approximates the perimeter and/or cross-section and/or surface areaand/or shape of the original virtually planned distal femoral bone cut.

For example, if the comparison of the perimeter and/or cross-sectionand/or surface area and/or shape of the physical cut distal femoral bonewith the perimeter and/or cross-section and/or surface area and/or shapeof the virtually planned cut distal femoral bone and, for example, theoptional identification of a new virtual cut plane that corresponds inthe virtual data to the physical distal femoral cut show that thedifference in location, position, orientation and/or angulation betweenthe virtual planned distal femoral cut and the physical femoral cutexceeds a threshold value, e.g. 3 degrees more angulation in flexiondirection and/or 2 mm greater cut depth (i.e. more bone removal), thenthe surgeon can modify the registration of the live data of the patient(e.g. the perimeter and/or cross-section and/or surface area and/orshape of the physical distal cut distal femoral bone) with the virtualdata of the patient by registering the corresponding virtual cut planewith the physical cut plane and the surgeon or the software can modifythe virtual surgical plan. The modifications to the virtual surgicalplan can, in this example, include that the angulation of the anteriorfemoral cut, posterior femoral cuts and the chamfer cuts will be changedto align with the distal femoral cut consistent with the dimensions andangulation of the planar surfaces of the femoral implant component inorder to avoid a gap between the implant and the bone or an area wherethe remaining physical cut bone is too wide, which may result in thebone being too wide in select areas, wider than the implant dimensionsthereby not accepting the implant. If a femur first technique is used,the modifications of the virtual surgical plan can also include that thecut height or depth of the proximal tibial cut and the cut angulation ofthe proximal tibial cut be adjusted, for example by cutting less tibiaand by changing the slope of the cut to account for a more flexedfemoral component and to maintain better soft-tissue/ligament balanceaccounting for the different physical distal femoral cut. Theseadjustments to the virtual surgical plan can optionally be displayed bythe OHMD, e.g. by displaying one or more virtual corrected or adjustedanterior, posterior, chamfer cuts, and/or by displaying one or morecorrected or adjusted proximal tibial cut(s) with a corrected oradjusted cut height/depth and/or corrected or adjusted tibial slopeand/or corrected or adjusted tibial varus or valgus angle. The OHMD candisplay the virtually corrected or adjusted intended/projected path ofthe saw blade or surgical instrument, the virtually corrected oradjusted intended/projected cut planes, or the virtually corrected oradjusted intended/projected axes of the saw blade and/or power tools.

The following is another example, where the surgeon inadvertentlymis-directs the femoral cut, with the assistance of the OHMD and anintegrated or attached or separate image and/or video capture systemdetects the femoral miscut and then decides to perform the necessarycorrection(s) in a subsequent surgical step on the tibial side.

FIG. 9A shows a predetermined distal femoral cut and proximal tibialcut, for example as part of a view of a virtual surgical plan, asoptionally displayed by OHMD in 2D or 3D, non-stereoscopic orstereoscopic. The OHMD can display a virtual predetermined path for aphysical saw blade selected to make the distal femoral cut 120 and theproximal tibial cut 121 in this example. The OHMD can also display thevirtual distal femoral and/or proximal tibial cut. The cut location canbe adjusted for the thickness of the saw blade.

The display of the predetermined path can be in 2D or in 3D,non-stereoscopic or stereoscopic. The surgeon can align the physical sawblade with the predetermined path and the surgeon can then advance thesaw blade while keeping the saw blade substantially aligned with thepredetermined path as shown by the OHMD. Rather than display thepredetermined path, the OHMD can also display a virtual bone saw alignedto make the virtual bone cut (optionally accounting for bone lost fromthe cutting or sawing) and the surgeon can align the physical bone sawwith the virtual bone saw and make the cut.

FIG. 9B shows a cross-section or view of the intended virtual distalfemoral cut 122, for example as developed in the virtual surgical plan.The perimeter and/or cross-section and/or surface area and/or shape ofthe virtually cut femur, for example simulated using data from apre-operative imaging study of the patient, e.g. CT or MRI or ultrasoundis visible.

FIG. 9C shows the physical distal femoral cut 123 made in the livepatient. The physical distal femoral cut 123 is not aligned with thevirtually predetermined path for the saw blade and it is not alignedwith the virtual distal femoral cut 120 in the virtual surgical plan inthis example. This can happen for various reasons in live surgery, forexample unexpectedly sclerotic areas of bone that cause saw bladedeviation. The difference in alignment between the virtually intendedbone cut and the physical femoral bone cut can be difficult to detectfor the surgeon intraoperatively. Broken line indicates predeterminedtibial cut based on virtual surgical plan.

FIG. 9D shows the view or cross-section of the physical distal femoralcut 124. The perimeter and/or cross-section and/or surface area and/orshape of the physical distal femoral cut is different than the perimeterand/or cross-section and/or surface area and/or shape of the virtuallyplanned cut femur 122. The perimeter and/or cross-section and/or surfacearea and/or shape of the physical cut distal femur can be detected, forexample using an image and/or video capture system integrated into,attached to or separate from the OHMD or using a laser scanner and/or 3Dscanner or using a mechanical or optical probe, for example using RF,optical, navigation and other markers. It can then be compared to theperimeter and/or cross-section and/or surface area and/or shape of thevirtual cut surface.

In FIG. 9E, once the perimeter and/or cross-section and/or surface areaand/or shape of the physical cut distal femur has been detected, forexample using an image and/or video capture system integrated into,attached to or separate from the OHMD or using a laser scanner and/or 3Dscanner or using a mechanical or optical probe, for example using RF,optical, navigation and other markers, a corresponding perimeter and/orcross-section and/or surface area and/or shape of the physical cutdistal femur can be identified in the virtual data of the patient 125,for example using image processing algorithms known in the art.

In FIG. 9F, once the corresponding perimeter and/or cross-section and/orsurface area and/or shape has been identified in the virtual data of thepatient (FIG. 9E), a new, substitute virtual femur cut can optionally beidentified 126 in the virtual data of the patient. The difference inposition, location, orientation, coronal, sagittal, axialangle/angulation between the originally planned or predetermined virtualfemoral bone cut 120 and the new, substitute femoral bone cut 126 can bedetermined. Depending on the severity and/or clinical significance ofthe difference between the originally planned or predetermined virtualfemoral bone cut and the physical femoral bone cut, the surgeon candecide or chose between or combine any of the preceding Options A-E,e.g. modify the last surgical step, modify the next surgical step(s),modify the virtual surgical plan of the patient, modify the registrationof the virtual data of the patient in relationship to the live data ofthe patient, and/or apply registration correction or combinationsthereof. In this example, the surgeon is electing to modify the nextsurgical step(s) by changing the angulation of the virtual tibial cut(s)from its original orientation 121 to a new orientation 127 that canoptionally, at least partially, correct the overall alignment for thefemoral mis-cut.

In FIG. 9G, the surgeon can elect to modify the next surgical step(s)and, in this example, change the proximal tibial cut by applying acorrection in the alignment and direction of the saw blade to execute onthe new, virtually modified tibial cut. The modified virtual and theresultant physical proximal tibial bone cut 128 can be placed to atleast partially correct for the femoral mis-cut.

if the comparison of the perimeter and/or cross-section and/or surfacearea and/or shape of the physical cut distal femoral bone with theperimeter and/or cross-section and/or surface area and/or shape of thevirtually planned cut distal femoral bone and, for example, the optionalidentification of a new virtual cut plane that corresponds in thevirtual data to the physical distal femoral cut shows that the physicaldistal femoral cut surface is more angled, e.g. 3 degrees or more, incoronal direction than intended in the virtual surgical plan and/or thevirtually cut distal femoral surface as displayed by the OHMD, then thesurgeon can modify the last surgical step by re-cutting the distalfemoral bone to correct the error in coronal plane angulation and toavoid any varus/valgus misalignment. The virtual predetermined cut planeor the virtual predetermined path for the saw blade or the virtualpredetermined axis of the saw blade and/or power instrument and/or thevirtual saw blade and/or power instrument aligned/oriented for thecorrection of the last surgical step can optionally be displayed by theOHMD. Alternatively, the surgeon can elect to correct one or more of thenext surgical step(s), e.g. in this example by changing the intended cutfor the tibial plateau to correct for the femoral cut coronal planemisangulation. The surgeon can align in either example the physical sawblade or surgical instrument with one or more of the virtualpredetermined cut plane or the virtual predetermined path for the sawblade or the virtual predetermined axis of the saw blade and/or powerinstrument and/or the virtual saw blade and/or power instrument.

FIG. 10A shows a predetermined distal femoral cut and proximal tibialcut, for example as part of a view of a virtual surgical plan, asoptionally displayed by OHMD in 2D or 3D, non-stereoscopic orstereoscopic. The OHMD can display a mechanical 130 or anatomicaxis/axes of the knee, e.g. a femoral axis or a tibial axis, as well asvarious other kinematic or biomechanical axes, including a rotation axisof the knee. The virtual surgical plan can include the planning offemoral 131 and/or tibial 132 bone cuts that can be selected to correctany underlying mechanical axis deformity, e.g. varus or valgusdeformity. For example, one or more of these bone cuts can be selectedto be perpendicular to the patient's femoral or tibial mechanical axis.Alternatively, other alignments can be chosen and can be incorporatedinto the virtual surgical plan. For example, the medial femoral condylesurface, lateral femoral condyle surface and the medial tibial surfaceand lateral tibial surface can be optionally aligned with the patient'scartilage and/or subchondral bone or subchondral bone with an offsetadded to account for lost cartilage. The OHMD can display one or morevirtual predetermined path (broken horizontal lines) for a physical sawblade selected to make the femoral cut and/or the tibial cut in thisexample. The OHMD can also display the virtual femoral and/or tibialcut. The virtual/projected path for a physical saw blade to make thefemoral and/or tibial cut and the virtual femoral and/or tibial cut canbe the same; they can also be different, for example accounting for thethickness of the saw blade. Rather than provide a virtual display of thepredetermined path or plane, the OHMD can also display a virtualrepresentation of a virtual bone saw or a 2D or 3D outline thereofaligned to make the virtual bone cuts (optionally accounting for bonelost from the cutting or sawing) and the surgeon can align the physicalbone saw with the virtual bone saw or its 2D or 3D outline and make thecut.

FIG. 10B shows a cross-section or view of the intended virtual femoralcut 133, for example as developed in the virtual surgical plan. Theperimeter and/or cross-section and/or surface area and/or shape of thevirtually cut femur, for example simulated using data from apre-operative imaging study of the patient, e.g. CT or MRI orultrasound, is relatively round in this example for the lateral condyle(left) and the medial condyle (right).

FIG. 10C shows the physical distal femoral cut made in the live patient134. The physical femoral cut is not aligned with the virtuallypredetermined path for the saw blade and it is not aligned with thevirtual femoral cut in the virtual surgical plan in this example. Thiscan happen for various reasons in live surgery, for example unexpectedlysclerotic areas of bone or soft bone or osteoporotic bone that cause sawblade deviation.

FIG. 10D shows the view or cross-section of the physical femoral cut135. The perimeter and/or cross-section and/or surface area and/or shapeof the physical femoral cut is different than the perimeter and/orcross-section and/or surface area and/or shape of the virtually plannedfemoral cut. The perimeter and/or cross-section and/or surface areaand/or shape of the physical cut distal femur can be detected, forexample using an image and/or video capture system integrated into,attached to or separate from the OHMD or using a laser scanner and/or 3Dscanner or using a mechanical or optical probe, for example using RF,optical, navigation and other markers. It can then be compared to theperimeter and/or cross-section and/or surface area and/or shape of thevirtual cut surface.

In FIG. 10E, once the perimeter and/or cross-section and/or surface areaand/or shape of the physical cut distal femur has been detected, forexample using an image and/or video capture system integrated into,attached to or separate from the OHMD or using a laser scanner and/or 3Dscanner or using a mechanical or optical probe, for example using RF,optical, navigation and other markers, a corresponding perimeter and/orcross-section and/or surface area and/or shape of the physical cutdistal femur can be identified in the virtual data of the patient 136,for example using image processing algorithms known in the art.

In FIG. 10F, once the corresponding perimeter and/or cross-sectionand/or surface area and/or shape has been identified in the virtual dataof the patient (FIG. 10E), a virtual femur cut 137 which approximatesthe physical femoral cut can be identified in the virtual data of thepatient. The difference in position, location, orientation, coronal,sagittal, axial angle/angulation between the originally planned orpredetermined virtual femoral bone cut 131 and the new virtual femoralbone cut and the physical bone cut can be determined. Depending on theseverity and/or clinical significance of the difference between them,the surgeon can then decide or chose between or combine any of thepreceding Options A-E, e.g. modify the last surgical step (e.g. recutthe femur), modify the next surgical step(s) (e.g. cut the tibia at adifferent coronal angulation than originally planned to account for thefemoral mis-cut and, optionally, to achieve a composite alignment thatis, for example, still within normal (180 degrees) mechanical axisalignment), modify the virtual surgical plan of the patient, modify theregistration of the virtual data of the patient in relationship to thelive data of the patient, and/or apply registration correction orcombinations thereof.

In FIG. 10G, the surgeon can elect to modify the next surgical step and,in this example, modify the proximal tibial cut as shown in the twoexamples, one with a straight broken dotted line 139 and the other witha straight dotted line 138. In some embodiments, the surgeon can cut thetibia at a different coronal angulation than originally planned toaccount for the femoral mis-cut and, optionally, to achieve a compositealignment that is, for example, still within normal (180 degrees)mechanical axis alignment.

In another example, an OHMD can be used for guiding the placement offemoral pins or drills, which can be utilized for setting femoralcomponent rotation, as is commonly done in total knee replacementprocedures. Such femoral pins or drills can, for example, be placedthrough openings in a femoral cut block or a pin or drill block. In thisexample, the OHMD can guide the placement of the physical femoral cutblock or pin or drill block by projecting a virtual femoral cut block orpin or drill block with which the surgeon can align the physical femoralcut block or drill or pin block, followed by the placement of thephysical pins or drills. Alternatively, the OHMD can guide the placementof the physical pins or drills by projecting the virtual pins or drillsor by projecting virtual pin or drill paths, followed by the placementof the physical pins or drills.

An image and/or video capture system integrated into, attached to orseparate from the OHMD, or an optical or mechanical probe, optionallywith attached optical markers, LEDs, navigation markers including, butnot limited to, infrared markers, retroreflective markers, RF markers,and/or IMUs, optical markers, LEDs, navigation markers including, butnot limited to, infrared markers, retroreflective markers, RF markers,and/or IMUs attached to the drills or pins can be used for assessing theposition and/or orientation and/or alignment of the one or more physicalpins or drills or resultant physical pin or drill holes and to comparethem to the position and/or orientation and/or alignment of the one ormore virtual pins or drills or virtual pin or drill holes, e.g. in thepatient's virtual surgical plan using, for example, the existingregistration or a new registration of the live and virtual data usingthe surgically altered or modified surface. If a difference in positionand/or orientation and/or alignment between the physical and the virtualpins or drills or pin holes or drill holes is detected and found to beclinically significant, the surgeon can then decide or chose between orcombine any of the preceding options A-E, e.g. modify the last surgicalstep (e.g. repeat/revise one or more pin placements), modify the nextsurgical step(s) (e.g. change femoral rotation to be different thanindicated by the one or more pins or drills or pin holes or drillholes), modify the virtual surgical plan of the patient, modify theregistration of the virtual data of the patient in relationship to thelive data of the patient, and/or apply registration correction orcombinations thereof.

In another example, an OHMD can be used for guiding the placement oftibial pins or drills, which can be utilized for setting tibialcomponent rotation, as is commonly done in total knee replacementprocedures. Such tibial pins or drills can, for example, be placedthrough openings in a tibial cut block or a pin or drill block. In thisexample, the OHMD can guide the placement of the physical tibial cutblock or pin or drill block by projecting a virtual tibial cut block orpin or drill block with which the surgeon can align the physical tibialcut block or drill or pin block, followed by the placement of thephysical pins or drills. Alternatively, the OHMD can guide the placementof the physical pins or drills by projecting the virtual pins or drillsor by projecting virtual pin or drill paths, followed by the placementof the physical pins or drills.

An image and/or video capture system integrated into, attached to orseparate from the OHMD, or an optical or mechanical probe, optionallywith attached RF markers, optical markers, LEDs, navigation markersincluding, but not limited to, infrared markers, retroreflectivemarkers, RF markers, and/or IMUs, or optical markers, LEDs navigationmarkers including, but not limited to, infrared markers, retroreflectivemarkers, RF markers, and/or IMUs attached to the drills or pins can beused for assessing the position and/or orientation and/or alignment ofthe one or more physical pins or drills or resultant physical pin ordrill holes and to compare them to the position and/or orientationand/or alignment of the one or more virtual pins or drills or virtualpin or drill holes, e.g. in the patient's virtual surgical plan using,for example, the existing registration or a new registration of the liveand virtual data using the surgically altered or modified surface. If adifference in position and/or orientation and/or alignment between thephysical and the virtual pins or drills or pin holes or drill holes isdetected and found to be clinically significant, the surgeon can thendecide or chose between or combine any of the preceding options A-E,e.g. modify the last surgical step (e.g. repeat/revise one or more pinplacements), modify the next surgical step(s) (e.g. change tibialrotation to be different than indicated by the one or more pins ordrills or pin holes or drill holes), modify the virtual surgical plan ofthe patient, modify the registration of the virtual data of the patientin relationship to the live data of the patient, and/or applyregistration correction or combinations thereof.

Similarly, if the surgeon mis-cut, e.g. overcut the tibia, the OHMD canproject optional modifications to the femoral cut, e.g. moving thevirtual femoral cut and resultant physical femoral cut more distal toaccount for a tibial over-resection.

The preceding examples are in no way meant to be limiting of theinvention, but are only exemplary of the invention. Someone skilled inthe art can readily recognize how they can be applied to other types ofsurgery, e.g. ankle replacement, shoulder replacement, elbowreplacement, ligament repair and/or reconstruction or replacement,spinal procedures, e.g. vertebroplasty, kyphoplasty, spinal fusionand/or pedicle screw and rod placement.

Pin Based Registration, Registration after Bone Cuts, Reaming, Milling,Etc.

If the tissue is being drilled or a pin or drill is placed in thetissue, for example for placement of a pedicle screw with pin placementor drilling through portions or all of the pedicle or for placement of acut block in partial or total knee replacement or for planning a femoralcut or acetabular reaming for hip arthroplasty or for shoulderarthroplasty or for various types of surgery, e.g. cranial/brainsurgery, the registration procedure can be repeated after the pin ordrill has been placed or after the drilling has occurred. For example,an initial registration can be performed using an intraoperative x-ray,e.g. of a spine, or a knee, or a hip, e.g. with the patient in a proneposition or supine position. The intraoperative x-ray can include one ormore of an AP projection, PA projection, lateral projection, e.g. fromleft and/or from right side, oblique views, CT view using rotationalx-ray acquisition, e.g. on rotating C-arm system. One or more of theintra-operative x-ray projections can be matched with pre-operativeimaging data of the patient or virtual data of the patient including,optionally, a virtual surgical plan, using, for example, patternrecognition algorithms, image processing algorithms, or manual/visualmatching by the surgeon or operator, optionally with magnificationadjustment for a given film/detector focus distance, with magnificationor de-magnification of either the intraoperative x-ray data, thepre-operative data, the virtual data of the patient, including,optionally, the virtual surgical plan, with the aim that all data usedhave similar or the same magnification.

In the example of spinal surgery, once the initial registration has beenperformed, a pin or drill can be placed in a first pedicle, e.g. in acervical, thoracic or lumbar spine. Then a second pin or drill and/oradditional pins or drills can be placed in a second pedicle, optionallyat the same or different spinal levels, optionally on the same side ofthe spine (e.g. left or right) or alternatingly left and right fromspinal level to spinal level. Similarly, pins or drills can be placedand registered for various aspects of knee replacement surgery, hipreplacement surgery, shoulder replacement surgery, ACL repair orreconstruction and/or various sports related surgeries and/orcranial/brain surgery.

The position of the one or more pins or drills can be registered, forexample using an image and/or video capture system integrated into,attached to or separate from the OHMD or using a laser scanner and/or 3Dscanner that detects the one or more pins or drills. The position of theone or more pins or drills can be registered using attached orintegrated optical markers or navigation markers including, but notlimited to infrared markers, retroreflective markers, RF markers, e.g.with an optionally used navigation system, or IMUs. The position of thedrill(s) or pin(s) can be detected using a touch probe, wherein thetouch probe can include attached or integrated IMUs, optical markers,navigation markers including, but not limited to, infrared markers,retroreflective markers, RF markers and the like, using for example animage and/or video capture system or a navigation system. If more thanone marker is placed along the trajectory of the pin or drill or ifimage capture is used, the two or more markers or the trajectory of thevisualized portions of the pin(s) or drill(s) using image capture can beused to estimate the trajectory of the pin(s) or drill(s) and toestimate a projected path as the pin(s) or drill(s) are advanced. If thelength and the thickness of the pins are known, not only the endpointoutside the patient's tissue can be determined, but also the location ofthe tip can be estimated even though it can be seated deep inside thepatient's tissue in spinal surgery, knee replacement, hip replacement,shoulder replacement, brain surgery and various types of other surgery.

The position of the pins or drills can be registered in relationship tothe patient and/or the OHMD using any of the techniques described in thespecification. The one or more optical markers can be retroreflective orcan include LEDs. Combinations of optical and RF markers can be used.

In some embodiments of the invention, a first drill or pin isregistered, optionally followed by registration of a second or more pinand drills. The position and/or orientation of the one or more pins ordrills can be used to maintain registration during the surgery, e.g.placement of pedicle screws and related devices, e.g. rods, or kneereplacement with placement of one or more pins or drills in the femurand/or the tibia or hip replacement with placement of one or more pinsor drills in the acetabulum or proximal femur. Since the one or morepins or drills are fixed to the bone, accurate registration can bemaintained even if there is patient movement after the initialregistration, if the pin or drill(s) are used for registration after theinitial registration. Optionally, both the initial registration and thesubsequent registration to the altered surgical surface/site after theplacement of the pin or drill with registration to the pin or drill(s)can be used together. In this case, statistical techniques can beapplied to reconcile small differences between the initial registrationand the registration to the altered surgical surface or site includingthe one or more pins or drills. For example, the mean or the median ofthe different registrations can be used for any subsequent surgicalsteps.

In some embodiments of the invention, an initial registration can beperformed between virtual data of the patient, e.g. pre-operativeimaging, including optionally a virtual surgical plan for the patient,and live data of the patient during surgery. The initial registrationcan, for example, be performed using intra-operative imaging, which canbe referenced to and registered with the live data of the patient. Anyother technique of registration described in the specification or knownin the art can be used for the initial registration. A first pin ordrill or a first set of pins or drills can be placed using the initialregistration of virtual data of the patient and live data of thepatient.

Following the placement of a first pin or drill or a first set of pinsor drills, intra-operative imaging can be repeated. In some embodimentsof the invention, intra-operative imaging is used for the initialregistration and the same intraoperative imaging modality and techniqueor similar intra-operative imaging modality or technique is used afterplacing the first pin or drill or the first set of pins or drills.Alternatively, a different intra-operative imaging modality is usedafter placing the first pin or drill or the first set of pins or drills.Intra-operative imaging modalities can include, for example, x-rays,e.g. AP, PA, lateral and oblique views, C-arm acquisition, optionallywith CT capability, CT scan or ultrasound scan or MRI scan or any otherimaging technique known in the art.

In some embodiments of the invention, after a first pin or drill or afirst set of pins or drills is placed, the accuracy of the placement canbe assessed. The accuracy of the placement can be assessed using, forexample, any of the following:

-   -   Intraoperative imaging, e.g. also if the initial registration        was performed without use of intraoperative imaging    -   Intraoperative imaging using the same or a different imaging        modality used for an initial registration (if applicable)    -   Image capture of the visible portions of the pin(s) or drill(s),        with optional projection/estimation of the location and/or        orientation of any non-visualized portions inside the patient's        tissue    -   Optical markers, navigation markers including, but not limited        to, infrared markers, retroreflective markers, RF markers, IMUs,        and any other electronic or optical or magnetic marker known in        the art, with optional projection/estimation of the location        and/or orientation of any non-visualized portions inside the        patient's tissue

Any deviations in the physical placement including the physical positionand/or the physical orientation of the pin(s) or drill(s) compared tothe intended position and/or intended orientation of the pin(s) ordrill(s) in the virtual surgical plan can be measured in this manner. Ifone or more of the pins show a deviation in physical vs. intendedvirtual position and/or orientation, the difference in coordinates canbe determined and a coordinate transfer or coordinate correction can beapplied for any subsequent registration that uses one or more of thepins or drills placed inside the patient's tissue. A coordinate transferor coordinate correction can be applied globally, e.g. to all pins ordrills placed using the same values. Alternatively, a coordinatetransfer or coordinate correction can be applied individually to eachpin or drill accounting for their specific deviation from physical vs.intended virtual placement/position/and/or orientation. The formerapproach can be more time efficient. The latter approach can be moreaccurate for any subsequent registrations. A coordinate transfer orcoordinate correction applied to each pin or drill individually usingdata on the amount of deviation/difference in coordinates betweenphysical placement/position/and/or orientation compared to intendedvirtual placement/position/and/or orientation based on the virtualsurgical plan can be particularly helpful in spinal surgery, when one ormore spinal segment can move in relationship to each other during thesurgery, e.g. if the surgeon has to adjust the position of the patienton the table. In this case, one or more pins or drills can optionally beplaced at more than one spinal level, for example all spinal levelsinvolved in the surgery, after the initial registration and the accuracyof the placement can be assessed using the foregoing techniques. Acoordinate transfer or coordinate correction can then optionally beapplied for more than one spinal level, e.g. all spinal levels involvedin the surgery, wherein the difference in physical vs. intended virtualplacement/position/and/or orientation of the pins or drills can be usedto improve the accuracy of any subsequent registration using the one ormore pins or drills for subsequent surgical steps for each spinal levelfor which the coordinate transfer or coordinate correction has beenapplied.

In the example of spinal surgery, one or more pedicle screws can beplaced, at the same spinal level or different spinal levels. Optionally,the accuracy of the physical placement/position and/or orientation ofeach pedicle screw can be assessed compared to the intended virtualplacement/position/and/or orientation in the virtual surgical plan usingany of the foregoing techniques. Optionally a coordinate transfer orcoordinate correction can be determined based on any deviations betweenphysical and intended virtual placement of the pedicle screw and thepedicle screw can be used for registration of the patient, the spine,and/or the OHMD during any subsequent surgical steps, e.g. placement ofadditional pedicle screws, e.g. at the same or other spinal levels, orplacement of one or more connectors or rods and the like.

During the placement of the pedicle screw, registration can bemaintained by referencing one or more of the pins or drills or pediclescrews placed in the pedicles at the same or adjacent spinal levels.

Similarly, in other surgical procedures, e.g. knee replacement, hipreplacement, shoulder replacement, ACL repair and reconstruction,cranial, maxillofacial and brain surgery, the physical position of anydrill, pin, instrument, implant, device or device component can bedetermined using any of the techniques described in the specificationand any deviations or differences between the physical and the intendedvirtual placement/position/and/or orientation can be determined. Thedifferences measured can be used to determine a coordinate transfer orcoordinate correction for any subsequent registrations for subsequentsurgical steps using now the one or more drill, pin, instrument,implant, device or device component as the registration reference ormarker.

By referencing a pin or drill that is fixed inside the bone or a hardtissue (following the first surgical alteration), it is possible tomaintain accurate registration, e.g. during pedicle screw placement,knee replacement, hip replacement, ACL repair and/or reconstruction,maxillofacial surgery, cranial and/or brain surgery.

In this case, the pinned or drilled tissue of the live patient orportions thereof can be matched to or superimposed and/or registeredwith the corresponding pinned or drilled tissue in the virtual surgicalplan. Once an adequate match of the live and virtual cut pinned ordrilled area has been obtained, registration can optionally be repeated.In some embodiments of the invention, the bone void or hole created byany pinning or drilling can be used for any subsequent registrations.Optionally, a pin or drill can be temporarily placed back into the bonevoid or hole for any subsequent registration and subsequent surgicalsteps. If other surgical instruments are used, e.g. other than a drillor pin, such as a burr or a blade, other resultant bone voids canoptionally also be used for any subsequent registrations.

Optionally, the position, location, and/or orientation and/or sizeand/or shape of any bone void or hole created by any surgical instrumentcan be assessed, e.g. using intraoperative imaging such as x-rays orultrasound, and the difference between the physical and the intendedvirtual position, location, and/or orientation and/or size and/or shapeof any bone void or hole can be assessed. The difference or deviationbetween the physical and the intended virtual position, location, and/ororientation and/or size and/or shape of the bone void or hole can beused to determine a coordinate difference or coordinate transfer orcoordinate correction so that the bone void or hole can be used for anysubsequent registration and subsequent surgical steps. Any subsequentregistration can be performed by optionally introducing a partial orcomplete bone void filler (e.g. a pin or a drill) and registering thebone void filler. Any subsequent registration can also be performed byregistering the bone void or hole directly, e.g. with intraoperativeimaging. Any subsequent registration can also be performed by placingone or more IMUs, optical markers, and/or navigation markers including,but not limited to, infrared markers, retroreflective markers, RFmarkers inside or adjacent to the bone void and registered one or moreof the IMUs, optical markers, LEDs and/or navigation markers including,but not limited to, infrared markers, retroreflective markers, RFmarkers using any of the techniques described in the specification.Moreover, any subsequent registration can also be performed by markingportions or all of the bone void or hole with a color, e.g. toluidineblue, and by registering the marked and/or stained portions of the bonevoid or hole, e.g. using an image and/or video capture system integratedinto, attached to, or separate from the OHMD.

If a tissue cut is performed, for example with a scalpel or a saw, theregistration procedure can be repeated after the tissue cut has beenplaced. In this case, the cut tissue surface of the live patient orportions thereof or the perimeter of the cut tissue surface of the livepatient or portions thereof or the surface area of the cut tissuesurface of the live patient or portions thereof or the volume of theremoved tissue of the live patient or portions thereof can be matched toor superimposed and/or registered with the corresponding cut tissuesurface of the virtual data or portions thereof or the perimeter of thecut tissue surface of the virtual data or portions thereof or thesurface area of the cut tissue surface of the virtual data or portionsthereof or the volume of the removed tissue of the virtual data orportions thereof in the virtual surgical plan. Once an adequate match ofthe live and virtual cut surfaces has been obtained, registration canoptionally be repeated.

If a tissue cut is performed, the registration procedure can be repeatedafter the tissue cut has been completed. In this case, the cut tissuesurface of the live patient or portions thereof or the perimeter of thecut tissue surface of the live patient or portions thereof can bematched to or superimposed onto and/or registered with the correspondingcut tissue surface or portions thereof in the virtual surgical plan orthe perimeter of the cut tissue surface in the virtual surgical plan orportions thereof. Once an adequate match of the live and virtual cutsurfaces has been obtained, registration can optionally be repeated.

If a bone cut is performed, for example with a saw, the registrationprocedure can be repeated after the bone cut has been placed. In thiscase, the cut bone surface of the live patient or portions thereof orthe perimeter of the cut bone surface of the live patient or portionsthereof or the surface area of the cut bone surface of the live patientor portions thereof or the volume of the removed bone of the livepatient or portions thereof can be matched to or superimposed ontoand/or registered with the corresponding cut bone surface of the virtualdata or portions thereof or the perimeter of the cut bone surface of thevirtual data or portions thereof or the surface area of the cut bonesurface of the virtual data or portions thereof or the volume of theremoved bone of the virtual data or portions thereof in the virtualsurgical plan. Once an adequate match of the live and virtual cutsurfaces has been obtained, registration can optionally be repeated.

If a milling, reaming or impacting procedure is performed, for examplewith a reamer, a mill or an impactor, the registration procedure can berepeated after the milling, reaming or impacting has been performed. Inthis case, the milled, reamed or impacted bone surface of the livepatient or portions thereof or the perimeter of the milled, reamed orimpacted bone surface of the live patient or portions thereof or thesurface area of the milled, reamed or impacted bone surface of the livepatient or portions thereof or the volume of the removed bone of thelive patient or portions thereof can be matched to or superimposed ontoand/or registered with the corresponding milled, reamed or impacted bonesurface of the virtual data or portions thereof or the perimeter of themilled, reamed or impacted bone surface of the virtual data or portionsthereof or the surface area of the milled, reamed or impacted bonesurface of the virtual data or portions thereof or the volume of theremoved bone of the virtual data or portions thereof in the virtualsurgical plan. Once an adequate match of the live and virtual cutsurfaces has been obtained, registration can optionally be repeated.

If a drilling procedure is performed, for example with a drill or a pinor a K-wire, the registration procedure can be repeated after the drillor pin or K-wire has been placed. In this case, the drilled surface ofthe live patient or portions thereof or the perimeter of the drilledsurface of the live patient or portions thereof or the surface area ofthe drilled surface of the live patient or portions thereof or thevolume of the removed bone of the live patient or portions thereof orthe location of the drill hole or the orientation of the drill hole orthe size of the drill hole or a marker such as a drill, a pin or aK-wire or an ink inserted into the drill hole can be matched to orsuperimposed onto and/or registered with the corresponding drilledsurface in the virtual data or portions thereof or the perimeter of thedrilled surface in the virtual data or portions thereof or the surfacearea of the drilled surface in the virtual data or portions thereof orthe volume of the removed bone in the virtual data or portions thereofor the location of the drill hole in the virtual data or the orientationof the drill hole in the virtual data or the size of the drill hole inthe virtual data or a marker such as a drill, a pin or a K-wire or anink inserted into the drill hole in the virtual data, optionally in thevirtual surgical plan. Once an adequate match of the live and virtualcut surfaces has been obtained, registration can optionally be repeated.

If a drilling procedure is performed, the drill holes can optionally bemarked with india ink or another color in the live patient. The colormarking can be recognized with use of an image and/or video capturesystem integrated into, attached to or separate from the OHMD. The colormarkings in the live patient can then optionally be used to re-registerthe live data of the patient with the virtual data after one or moresurgical alterations of the tissue has/have been performed. The colormarkings can be used with an image and/or video capture system to detectthem in the live patient data and to register them with the virtualpatient data. Alternatively, the color markings can be used by thesurgeon to identify the previously placed drill holes visually, forexample after one or more surgical alterations or surgical steps havebeen performed. A drill, a pin, a K-wire, a screw, or another surgicalinstrument can then optionally be placed inside the drill hole and theregistration of the live data and the virtual data can be performed bymatching, superimposing and/or registering the live drill, pin, K-wire,screw, or other surgical instrument with a corresponding virtual drill,pin, K-wire, screw, or other surgical instrument or a correspondingdrill hole in the virtual surgical plan.

For example, in a knee replacement procedure, a drill guide can beapplied to the distal femur and/or the distal femoral condyles beforethe distal femoral cut and bone removal is performed. The drill guidecan be integrated into the distal femoral cut block. Typically, two ormore drill holes can be placed, for example with one or more drill holeslocated in the medial femoral condyle or in the medial femur and one ormore drill holes located in the lateral femoral condyle or in thelateral femur. The location of the medial and lateral drill holes andthe intersect between the two drill holes can be used to define therotation axis of the femoral component.

The OHMD can display the desired location of the distal femoral cutblock for achieving the desired mechanical axis correction and thedesired location of the drill holes for setting the desired rotationaxis of the femoral implant component. The drill holes can be drilledprior to performing the cut and can be optionally marked with ink priorto performing the distal femoral cut. The distal femoral cut can then beperformed. The ink in the drill holes can then be identified on the cutsurface. The ink seen in the live patient data can be registered usingan image and/or video capture system integrated into, attached to orseparate from the OHMD and can be registered in relationship to thevirtual drill holes as defined in the virtual surgical plan.Alternatively, the surgeon can elect to insert a drill, pin, K-wire,screw, or other surgical instrument into the drill holes in the livepatient data and the location of the drill, pin, K-wire, screw, or othersurgical instrument can be registered using an image and/or videocapture system integrated into, attached to or separate from the OHMDand can be registered in relationship to a virtual drill, pin, K-wire,screw or other surgical instrument optionally introduced into thevirtual surgical plan.

In this manner, live patient data and virtual patient data can bere-registered after the distal femoral bone cut has been performed. Thesurgeon can also use the re-registration to check the accuracy of theinitial registration and perform adjustments to the physical surgicalplan or the virtual surgical plan depending on any discrepanciesdetected.

The foregoing embodiment can be applied to any type of joint replacementor joint sparing procedure including arthroscopy.

If a radiofrequency ablation, heat ablation, cryoablation, orcauterization is performed, the registration procedure can be repeatedafter the radiofrequency ablation, heat ablation, cryoablation, orcauterization has been performed. In this case, the ablated orcauterized tissue surface of the live patient or portions thereof or theperimeter of the ablated or cauterized tissue surface of the livepatient or portions thereof or the surface area of the ablated orcauterized tissue surface of the live patient or portions thereof or thevolume of the removed tissue of the live patient or portions thereof canbe matched to or superimposed and/or registered with the correspondingablated or cauterized tissue surface of the virtual data or portionsthereof or the perimeter of the ablated or cauterized tissue surface ofthe virtual data or portions thereof or the surface area of the ablatedor cauterized tissue surface of the virtual data or portions thereof orthe volume of the removed tissue of the virtual data or portions thereofin the virtual surgical plan. Once an adequate match of the live andvirtual ablated or cauterized surfaces has been obtained, registrationcan optionally be repeated.

If a placement of a medical implant component, a trial implant, a tissuegraft, a tissue matrix, a transplant, a catheter, a surgical instrumentor an injection of cells or a drug is performed, the registrationprocedure can be repeated after the surgical step or surgical alterationhas been performed. In this case, the altered tissue of the live patientor portions thereof, the altered tissue surface of the live patient orportions thereof, or the perimeter of the altered tissue surface of thelive patient or portions thereof, or the surface area of the alteredtissue surface of the live patient or portions thereof, or the volume ofthe removed tissue of the live patient or portions thereof can bematched to or superimposed and/or registered with the correspondingaltered tissue of the virtual data or portions thereof, altered tissuesurface of the virtual data or portions thereof, or the perimeter of thealtered tissue surface of the virtual data or portions thereof, or thesurface area of the altered tissue surface of the virtual data orportions thereof, or the volume of the removed tissue of the virtualdata or portions thereof in the virtual surgical plan. Once an adequatematch of the live and virtual altered tissue has been obtained,registration can optionally be repeated.

Libraries of Surgical Instruments

In some aspects, the system includes libraries of surgical instrumentsfor different surgical procedures. The concept of a virtual library ofsurgical instruments used in a virtual surgical plan and optionallydisplayed by an OHMD during the live surgery, e.g. superimposed onto thephysical surgical instruments to provide positional, orientation ordirectional guidance of the physical surgical instrument according tothe virtual and/or intended surgical plan, is applicable to any surgicalprocedure, e.g. cardiovascular procedures, thoracic or pulmonaryprocedures, neurological procedures, urological procedures,gynecological procedures, hepatic or other inner organ procedures,intestinal procedures and/or musculoskeletal procedures. Virtual andphysical surgical instruments and implant components can be registeredin a common coordinate system, for example with one or more OHMDs andlive data of the patient; the OHMD can project or display a virtualrepresentation of the virtual surgical instrument.

In some embodiments, a virtual library of surgical instruments cancorrespond to a physical library of surgical instruments during surgery.Optionally, only a few, select surgical instruments can be included inthe virtual library of surgical instruments. These few select surgicalinstruments can, for example, be the ones used for the principal, keysurgical steps, or select sub-steps. Alternatively, all surgicalinstruments used during the live surgery can be included in a virtuallibrary of virtual surgical instruments.

The virtual library of virtual surgical instruments can include theseinstruments in various file formats. In some embodiments, CAD fileformats can be used. In general, any type of surface representation, 2Dor 3D shape representation 3D volume representation, 3D display anddifferent file formats can be used in a virtual surgical plan, followedby optional display by the OHMD during surgery.

Examples of libraries of surgical instruments that can be used inpedicle screw placement or spinal rod placement, artificial diskreplacement, hip replacement and knee replacement are provided below.Any other surgical instruments used in any other surgical procedure canbe utilized in a virtual surgical plan and/or can be displayed by theOHMD.

Pedicle Screw & Spinal Rod Placement

A virtual and/or physical library of surgical instruments for pediclescrew instrumentation and/or spinal rod placement can for exampleinclude:

For Pedicle Preparation:

-   -   Awl, e.g. round awl    -   Single ended feeler probe    -   Dual ended feeler probe    -   Sounding/feeler probe    -   Thoracic ball handle probe    -   Lumbar ball handle probe    -   Straight probe, e.g. lumbar, thoracic, cervical    -   Curved probe, e.g. lumbar, thoracic, cervical    -   Ratcheting handle    -   Taps of different diameter/dimensions

For Screw Insertion:

-   -   Screw driver, e.g.        -   Multi-axial screw driver        -   Self-retaining screw driver    -   Rod template    -   Rod inserter    -   Rod gripper    -   Bender, e.g. French bender    -   Single ended plug starter    -   Dual ended plug starter    -   Provisional driver

For Rod Reduction:

-   -   Compressor, e.g. parallel compressor    -   Distractor, e.g. parallel distractor

For Tightening:

-   -   Break-off driver, e.g. self-retaining    -   Obturator    -   Counter torque

Other Instruments:

-   -   Plug starter, e.g. non-break-off    -   Quick connector    -   Torque limiting driver    -   Tissue retractors    -   Frame to hold tissue retractors    -   Clamps

Plate Instruments:

-   -   Implant positioners    -   Screw driver, e.g. torque limiting or non-torque limiting    -   Measuring caliper    -   Measuring credit card    -   Counter torque    -   Plate holder, e.g. in line    -   Plate bender(s)    -   Forceps plate holder    -   Removal driver, e.g. hex head shaft style

The foregoing list of surgical instruments for pedicle screwinstrumentation and/or spinal rod placement is only an example. It is byno means meant to be limiting of the invention. Any current and futuresurgical instrument for pedicle screw instrumentation and/or spinal rodplacement can be used in a virtual surgical plan and live surgical planfor pedicle screw instrumentation and/or spinal rod placement.

All of the above surgical instruments can be provided in different sizesand/or diameters and/or widths and/or lengths and/or shapes and/ordimensions, for example based on the size or dimensions of the physicalimplant, implant component and/or medical device used.

Libraries of Medical Devices, Implants, Implant Components

Pedicle Screw & Spinal Rod Placement

A library of virtual and physical implants, implant components and/ormedical devices for pedicle screw instrumentation and/or spinal rodplacement can, for example, include screws including, but not limitedto, screw heads, screw thread portion, multi-axial screws, single-axialscrews, set screws, all of the foregoing in different sizes and/ordiameters (optionally color coded during the display in the OHMD);plates including, but not limited to, fixed plates, cross-link plates,multi-span plates, all of the foregoing in different sizes and/ordiameters (optionally color coded during the display in the OHMD); rodsincluding, but not limited to, straight rods, contoured rods, all of theforegoing in different sizes and/or diameters (optionally color codedduring the display in the OHMD). All of the foregoing device, devicecomponents, implants and implant components can be provided in differentdiameters, widths, lengths, dimensions, shapes, or sizes.

Knee Replacement

A library of virtual and physical implants, implant components and/ormedical devices for partial and total knee replacement can, for example,include left and right femoral components of different sizes, e.g. size1, 2, 3, 4, . . . , 17, 18, 19, 20, and shapes, e.g. without or withdistal medial-lateral femoral offset, e.g. 1, 2, 3, 4, 5, 6, 7, 8 ormore mm, without or with posterior medial-lateral femoral condyleoffset, e.g. 1, 2, 3, 4, 5, 6, 7, 8 or more mm; left and right tibialcomponents of different sizes, metal-backed or all polyethylene, e.g.size 1, 2, 3, 4, . . . , 17, 18, 19, 20, and shapes, e.g. symmetric,asymmetric, optionally with different degrees of asymmetry; left andright tibial inserts of different sizes, e.g. size 1, 2, 3, 4, . . . ,17, 18, 19, 20, and shapes, e.g. symmetric, asymmetric, optionally withdifferent degrees of asymmetry; left and right patellar components ofdifferent of different sizes, e.g. size 1, 2, 3, 4, . . . , 17, 18, 19,20, and shapes, e.g. symmetric, asymmetric.

Hip Replacement

A library of virtual and physical implants, implant components and/ormedical devices for hip replacement can, for example, include left andright standard offset, high offset, coxa vara offset femoral components,with collar or collarless, cemented or non-cemented, with differentporous ingrowth options, with different sizes, stem lengths, offsets,neck lengths, neck shaft angles; ceramic or metal femoral heads ofdifferent sizes, plus and minus heads; acetabular cups of differentsizes, cemented or non-cemented with different porous ingrowth options;different acetabular liners including lipped and asymmetric liners, ofdifferent sizes.

The foregoing lists are only of illustrative and exemplary nature andshould not be construed as limiting the invention. Any implant componentknown in the art can be included in one or more libraries of virtual andphysical implants.

Virtually Aligning Implant Components

An optical head mounted display can display or project digital hologramsof one or more virtual implants, virtual implant components and/orvirtual medical devices and virtual instruments without the use ofpre-operative or intra-operative imaging. In some embodiments of theinvention, the OHMD can display an arbitrary virtual implant componentover the surgical field. The arbitrary virtual implant component can,for example, be an implant component selected from the middle of a sizerange or a shape range. The arbitrary virtual implant component can beselected based on surgeon preferences. The arbitrary virtual implantcomponent can be the most common size used in a particular patientpopulation. The arbitrary virtual implant component can be moveableusing a virtual or other interface. For example, the virtualrepresentation of the arbitrary virtual implant component can include a“touch area”, wherein gesture recognition software, for example the oneprovided by Microsoft with the Microsoft Hololens including, forexample, the integrated virtual “drag function” for holograms can beused to move the arbitrary virtual implant component. For example, oneor more cameras integrated or attached to the OHMD can capture themovement of the surgeon's finger(s) in relationship to the touch area;using gesture tracking software, the arbitrary virtual implant componentcan then be moved by advancing the finger towards the touch area in adesired direction. A surgeon can, for example, also “hold” the arbitraryvirtual implant component by closing two fingers, e.g. thumb and indexfinger, over the touch area and then moving the fingers in the desireddirection, thereby moving the arbitrary virtual implant component intothe desired position and/or orientation on the patient's joint.

The OHMD can display the virtual implant component in any locationinitially, e.g. projected onto or outside the surgical field, e.g. a hipjoint, knee joint, shoulder joint, ankle joint, or a spine. The OHMD canoptionally display the virtual implant component at a defined angle,e.g. orthogonal or parallel, relative to a fixed structure in theoperating room, which can, for example, be recognized using one or morecameras, image capture or video capture systems integrated into the OHMDand spatial recognition software such as the one provided by Microsoftwith the Microsoft Hololens or which can be recognized using one or moreattached optical markers or navigation markers including, but notlimited to, infrared or RF markers. For example, one or more opticalmarkers can be attached to an extension of the operating table. The OHMDcan detect these one or more optical markers and determine theircoordinates and, with that, the horizontal plane of the operating roomtable. The virtual implant component can then be displayed perpendicularor at another angle relative to the operating room table. The virtualimplant component can be displayed at a defined angle to one or moreanatomic or biomechanical axes, e.g. a mechanical axis when a kneereplacement is contemplated. The virtual implant component can bedisplayed or projected tangent with one or more anatomic landmarks. Thevirtual implant component can be displayed intersecting one or moreanatomic landmarks.

The surgeon can move the virtual implant component to align it in thedesired location and/or orientation over the implantation site. Thesurgeon can then evaluate the size of the virtual implant component andthe fit of the virtual implant component by evaluating the size and fitof the virtual representation of the implant component superimposed ontothe intended implantation site. The surgeon can move and align thevirtual implant component so that, for example, its external surfaceco-locates, e.g. has similar or substantially the same coordinates, asthe external surface of the intended implantation site. The OHMD candisplay the other portions of the virtual implant component whichproject underneath the external surface of the implantation site. If thevirtual implant component is too large for an implantation site, thesurgeon can cancel the virtual display of the particular size of virtualimplant component displayed and the surgeon can select a smaller virtualimplant component from the library of virtual and physical implantcomponents. If the virtual implant component is too small for animplantation site, the surgeon can cancel the virtual display of theparticular size of virtual implant component displayed and the surgeoncan select a larger virtual implant component from the library ofvirtual and physical implant components. In this manner, the surgeon canoptimize the implant size and fit in three-dimensions in the actualsurgical site, rather than reverting to pre-operative sizing and fittingusing, for example, 2D xrays or 3D imaging studies, e.g. CT and MRI. Ifan implantation site is characterized by one or more asymmetries, e.g.in a knee joint or a tumor or an internal organ, the surgeon canoptionally size and fit one or more asymmetric implant components,optionally with different asymmetries and geometries, for theimplantation site.

The surgeon can move the virtual implant component to place it and/oralign and/it or orient in a desired position, location, and/ororientation over the implantation site for a given patient. Since themoving and aligning is performed over the live implantation site of thepatient, the surgeon can optimize the implant position, location, and/ororientation. The surgeon can further modify and/or optimize theposition, location, and/or orientation of the virtual implant componentand, with that, the physical implant component for a desired function inan implantation site, e.g. a desired flexion angle, rotation angle,range of motion, ligamentous laxity, desired movement. The surgeon canalign at least a portion of the external surface of the virtual implantcomponent with at least a portion of the external surface of theimplantation site. After the surgeon has placed, aligned and/or orientedthe virtual implant component superimposed in the desired positionand/or orientation over or aligned with the live implantation site, thecoordinates of the virtual implant component can be saved, e.g. in acommon coordinate system in which the OHMD and the implantation site canalso be registered. The saved coordinates of the virtual implantcomponent can, optionally be incorporated in a virtual surgical plan,which can optionally also be registered in the common coordinate system.The OHMD can subsequently display one or more digital holograms of oneor more virtual surgical instruments and/or virtual implant componentswherein the position, location, and/or orientation of the one or moredigital holograms of the one or more virtual surgical instruments and/orvirtual implant components are derived from or take into considerationthe saved coordinates of the virtual implant component.

For example, in a hip replacement, a virtual acetabular cup can bedisplayed near the surgical site including the exposed acetabulum of apatient. The surgeon can move the virtual acetabular cup using a virtualor other interface and superimpose it onto the patient's exposedacetabulum. The surgeon can evaluate the size and fit of the virtualacetabular cup. The surgeon can upsize or downsize the virtualacetabular cup by selecting smaller or larger virtual acetabular cupsuntil the surgeon is satisfied with the fit of the virtualrepresentation of the acetabular cup and the patient's exposedacetabulum. The surgeon can optionally center the virtual acetabular cupover the center of the patient's exposed acetabulum, matching the outerrim of the virtual acetabular cup to co-incide with or be equidistantsuperiorly, inferiorly, medially and laterally to the acetabular rim ofthe patient's exposed acetabulum. The coordinates of the virtualacetabular cup can then be saved, e.g. in the same coordinate system inwhich the surgical site, e.g. the acetabulum and/or the proximal femur,and the OHMD are registered. The coordinates of the virtual acetabularcup identified in this manner can be used to set a desired acetabularanteversion, e.g. during a reaming or impacting of the acetabular cup.Optionally, the virtual representation of the virtual acetabular cupfitted and placed by the surgeon can be displayed by the OHMD prior toimpacting the physical acetabular cup. The surgeon can then align thephysical acetabular cup with the virtual projection of the acetabularcup; once the desired alignment has been achieved, the surgeon can startimpact the physical acetabular cup, while optionally intermittentlycomparing its position and/or orientation including offset andanteversion with the virtual display of the virtual acetabular cup.

In some embodiments, in a hip replacement, a virtual femoral component,optionally including a head component, can be displayed near thesurgical site including the exposed proximal femur of a patient. Thesurgeon can move the virtual femoral component, optionally including ahead component, using a virtual or other interface and superimpose itonto the patient's exposed proximal femur, optionally before and/orafter the femoral neck cut. The surgeon can evaluate the size and fit ofthe virtual femoral component, optionally including a head component;optionally, the OHMD can display one or more pre-operative orintra-operative x-ray images or other imaging study, e.g. CT or MRI, ofthe patient registered in a common coordinate system with the surgicalsite; the imaging study can be superimposed onto the correspondingportions of the proximal femur, e.g. greater trochanter of live patientwith greater trochanter on x-ray or imaging study, lesser trochanter oflive patient with lesser trochanter on x-ray or image study etc. Thesurgeon can upsize or downsize the virtual femoral component byselecting smaller or larger virtual femoral components until the surgeonis satisfied with the fit of the virtual representation of theacetabular cup and the patient's exposed proximal femur or with the fitof the virtual representation of the femoral component and/or femoralhead and the patient's projected or displayed x-ray or imaging study,including marrow cavity and/or endosteal interface. The surgeon canoptionally center the virtual femoral component over the exposedproximal femur, optionally before and/or after the femoral neck cut,centering also over the cut femoral neck surface, aligning the virtualfemoral component and/or head with the corresponding anatomy or imagingdata of the patient. The coordinates of the virtual femoral componentand/or head can then be saved, e.g. in the same coordinate system inwhich the surgical site, e.g. the acetabulum and/or the proximal femur,and the OHMD are registered. The coordinates of the virtual femoralcomponent and/or head identified in this manner can be used to set adesired femoral anteversion and/or offset, e.g. during a reaming orbroaching of the femoral component. Optionally, the virtualrepresentation of the virtual femoral component and/or head fitted andplaced by the surgeon can be displayed by the OHMD prior to impactingthe physical femoral component. The surgeon can then align the physicalvirtual femoral component and/or head with the virtual projection of thefemoral component and/or head; once the desired alignment has beenachieved, the surgeon can start impact the physical femoral component,while optionally intermittently comparing its position and/ororientation including offset and anteversion with the virtual display ofthe virtual femoral component.

In some embodiments, in a knee replacement, a virtual femoral componentcan be displayed near the surgical site including the exposed distalfemur of a patient. The surgeon can move the virtual femoral component,using a virtual or other interface, e.g. a “touch zone” on the virtualrepresentation of the virtual femoral component with image or videocapture of the surgeon's hand and/or fingers and/or gesture tracking,and superimpose it onto the patient's exposed distal femur, optionallybefore and/or after any bone cuts. The surgeon can evaluate the size andfit of the virtual femoral component. The surgeon can evaluate the fitin three dimensions, anteriorly, posteriorly, at the medial aspect ofthe medial condyle, at the lateral aspect of the medial condyle, at themedial aspect of the lateral condyle, at the lateral aspect of thelateral condyle, in the intercondylar notch, in the medial and lateraltrochlear region. The surgeon can evaluate the size and fit of thevirtual femoral component for different degrees of femoral componentflexion and/or extension relative to the physical distal femur of thepatient and different degrees of femoral component rotation, e.g.external rotation. The surgeon can upsize or downsize the virtualfemoral component by selecting smaller or larger virtual femoralcomponents from the virtual library until the surgeon is satisfied withthe fit of the virtual representation of the femoral component and thepatient's exposed distal femur. If the virtual femoral implant componentis too large for an implantation site, the surgeon can cancel thevirtual display of the particular size of virtual femoral componentdisplayed and the surgeon can select a smaller virtual femoral componentfrom the library of virtual and physical femoral components. If thefemoral implant component is too small for an implantation site, thesurgeon can cancel the virtual display of the particular size of virtualfemoral component displayed and the surgeon can select a larger virtualfemoral component from the library of virtual and physical femoralcomponents. The surgeon can also evaluate the position and/ororientation of the virtual femoral component for possible notchingrelative to the physical anterior cortex of the distal femur of thepatient.

The surgeon can evaluate the shape of the virtual femoral component andcompare it with the shape of the patient's distal femur. The surgeon canoptionally align at least portions of the external surface of thevirtual femoral component with at least portions of the patient'sarticular surface, e.g. on the medial femoral condyle, the lateralfemoral condyle and/or the trochlear articular surface. The surgeon canselect different shapes of virtual femoral components from the virtuallibrary of implants, e.g. femoral components with one or more offsetsbetween the medial distal femoral condyle and the lateral distal femoralcondyle and/or one or more offsets, the same or different, between themedial posterior femoral condyle and the lateral posterior femoralcondyle. The offset can be a reflection of different radii of the medialdistal and/or posterior and the lateral distal and/or posterior femoralcondyle. For example, the surgeon can align at least portions of theexternal surface or projection of the medial condyle of the virtualfemoral component with at least portions of the external surface of thephysical medial condyle of the patient; if the external surface orprojection of the lateral condyle of the virtual femoral component isproud relative to the external surface of the physical lateral condyleof the patient, i.e. extends beyond the external surface of the physicallateral condyle of the patient, the surgeon can discard the digitalhologram of the virtual femoral component and select a different virtualfemoral component from the virtual library; for example, the surgeon canselect a virtual femoral component with a smaller lateral condyle radiusthan medial condyle radius and/or with a distal and/or posterior offsetof the lateral condyle compared to the medial condyle. The surgeon canproject, move, align, e.g. with the external surface of the medialand/or the lateral femoral condyle, multiple different virtual femoralcomponent shapes, e.g. with multiple different offsets, until thesurgeon has identified a virtual femoral component that yields thedesired shape. If a virtual femoral component is chosen with an offsetbetween the medial and the lateral femoral component, a matching offsetcan optionally be selected for the tibial polyethylene, wherein thelateral portion of the tibial insert can be 1, 2, 3, 4, 5, 6, 7, 8 ormore mm thicker than the medial portion of the tibial insert,corresponding to the smaller radius of the lateral femoral condyle ofthe virtual femoral component. The coordinates of the final position ofthe virtual femoral component can be saved and can, optionally, beincorporated into a virtual surgical plan. If the virtual surgical planindicates a variation in position, orientation, alignment, implantflexion of the virtual femoral component relative to the virtualsurgical plan, the surgeon can adjust the position of the virtualfemoral component to come closer to the intended position, orientation,alignment, implant flexion of the virtual surgical plan or to replicateit. Alternatively, the virtual surgical plan can be modified based onthe position, orientation, alignment, implant flexion of the virtualfemoral component.

In some embodiments, in a knee replacement, a virtual tibial componentcan be displayed near the surgical site including the exposed proximaltibia of a patient. The surgeon can move the virtual tibial component,using a virtual or other interface, e.g. a “touch zone” on the virtualrepresentation of the virtual tibial component with image or videocapture of the surgeon's hand and/or fingers and/or gesture tracking,and superimpose it onto the patient's exposed proximal tibia, optionallybefore and/or after any bone cuts. The surgeon can evaluate the size andfit of the virtual tibial component. The surgeon can evaluate the fit inthree dimensions, anteriorly, posteriorly, at the medial aspect of themedial tibial plateau, at the lateral aspect of the lateral tibialplateau. The surgeon can evaluate the size and fit of the virtual tibialcomponent for different levels of tibial resection and different tibialslopes and different degrees of tibial component rotation, e.g. externalrotation. The surgeon can upsize or downsize the virtual tibialcomponent by selecting smaller or larger virtual tibial components fromthe virtual library until the surgeon is satisfied with the fit of thevirtual representation of the tibial component and the patient's exposedproximal tibia. If the virtual tibial implant component is too large foran implantation site, the surgeon can cancel the virtual display of theparticular size of virtual tibial component displayed and the surgeoncan select a smaller virtual tibial component from the library ofvirtual and physical tibial components. If the tibial implant componentis too small for an implantation site, the surgeon can cancel thevirtual display of the particular size of virtual tibial componentdisplayed and the surgeon can select a larger virtual tibial componentfrom the library of virtual and physical tibial components. The surgeoncan also evaluate the position and/or orientation of the virtual tibialcomponent for possible PCL impingement with cruciate retaining implantsor patellar tendon impingement.

The surgeon can evaluate the shape of the virtual tibial component andcompare it with the shape of the patient's proximal tibia. The surgeoncan optionally select asymmetric virtual tibial components from anoptional variety of different asymmetric tibial shapes from the virtuallibrary of tibial components.

The surgeon can optionally align at least portions of the externalsurface of the virtual tibial component, e.g. the superior surface ofone or more polyethylene inserts with at least portions of the patient'stibial articular surface, e.g. on the medial tibial plateau, the lateraltibial plateau. By aligning at least portions of the external surface ofthe virtual tibial component, e.g. the superior surface of one or morepolyethylene inserts with at least portions of the patient's tibialarticular surface, e.g. on the medial tibial plateau, the lateral tibialplateau, the surgeon can determine the desired slope of the tibialresection, for example if the surgeon intends to cut the tibia andinstall the tibial component with a slope similar to the patient'snative slope. By aligning at least portions of the external surface ofthe virtual tibial component, e.g. the superior surface of one or morepolyethylene inserts with at least portions of the patient's tibialarticular surface, e.g. on the medial tibial plateau, the lateral tibialplateau, the surgeon can determine any desired medial to lateral offsetsfor the tibial polyethylene.

For example, the surgeon can align at least portions of the external,superior surface or projection of the medial portion of the virtualtibial component including the medial polyethylene with at leastportions of the external surface of the physical medial tibial plateauof the patient; if the external surface or projection of the external,superior surface of the lateral portion of the tibial polyethylene ofthe virtual tibial component is subjacent, inferior relative to theexternal surface of the physical lateral tibial plateau of the patient,i.e. remains below the external surface of the physical lateral tibialplateau of the patient, the surgeon can discard the digital hologram ofthe virtual tibial component and select a different virtual tibialcomponent from the virtual library; for example, the surgeon can selecta virtual tibial component including a polyethylene with a thickerlateral insert portion than medial insert portion. The surgeon canrepeat this process until a desired alignment, match or fit is achieved.The aligning of the external contour, shape or surface of the digitalhologram of the virtual tibial component medially and/or laterally withthe tibial plateau of the patient can take any desired varus or valguscorrection and/or slope into account, for example by adjusting aselected medial or lateral polyethylene thickness or shape based on thedesired varus or valgus correction and/or slope. The coordinates of thefinal position of the virtual tibial component can be saved and can,optionally, be incorporated into a virtual surgical plan. If the virtualsurgical plan indicates a variation in position, orientation, alignment,or slope of the virtual tibial component relative to the virtualsurgical plan, the surgeon can adjust the position of the virtual tibialcomponent to come closer to the intended position, orientation,alignment, and/or slope of the virtual surgical plan or to replicate it.Alternatively, the virtual surgical plan can be modified based on theposition, orientation, alignment, and/or slope of the virtual tibialcomponent.

Alignment criteria can be displayed by the OHMD while the surgeon ismoving, orienting or aligning a virtual femoral component, a virtualtibial component and/or a virtual patellar component. The resultantvarus/valgus, external/internal rotation, flexion of a femoral componentand/or slope of a tibial component, Q-angle and axes can be numericallyor graphically displayed and, optionally, compared, for example, withthe desired varus/valgus, external/internal rotation, flexion of afemoral component and/or slope of a tibial component based on a virtualsurgical plan. The surgeon can elect to apply different alignmentcriteria, for example anatomic alignment wherein the surgeon can, forexample, more closely match one or more virtual and physical implantsurfaces with one or more articular surfaces of the patient, e.g. on oneor two femoral condyles, on a medial and/or lateral tibial plateau, on atrochlea and/or a patella.

Someone skilled in the art can recognize that the foregoing embodimentscan be modified and applied to patellar replacement, patellarresurfacing, shoulder replacement, and/or ankle replacement.

In some embodiments of the invention, an intra-operative 2D or 3Dimaging study can be performed, e.g. one or more x-rays or a CT scan,for example using an O-arm system in spinal surgery. The intra-operativeimaging study can be registered in a common coordinate system with thesurgical site, e.g. a spine, and one or more OHMDs, for example worn bya first surgeon, a surgical resident and a physician assistant or anurse. The OHMD can display one or more digital holograms of subsurfaceanatomy of the patient, hidden or obscured by overlying skin,soft-tissue and/or bone. The OHMD can display an arbitrary virtualpedicle screw over the surgical field. The arbitrary virtual pediclescrew can, for example, be pedicle screw selected from the middle of asize range or a shape range. The arbitrary virtual pedicle screw can beselected based on surgeon preferences. The arbitrary virtual pediclescrew can be the most common size used in a particular patientpopulation. The arbitrary virtual pedicle screw can be moveable using avirtual or other interface. For example, the virtual representation ofthe arbitrary virtual pedicle screw can include a “touch area”, whereingesture recognition software, for example the one provided by Microsoftwith the Microsoft Hololens including, for example, the integratedvirtual “drag function” for holograms can be used to move the arbitraryvirtual pedicle screw. For example, one or more cameras integrated orattached to the OHMD can capture the movement of the surgeon's finger(s)in relationship to the touch area; using gesture tracking software, thearbitrary virtual pedicle screw can then be moved by advancing thefinger towards the touch area in a desired direction. A surgeon can, forexample, also “hold” the arbitrary virtual pedicle screw by closing twofingers, e.g. thumb and index finger, over the touch area and thenmoving the fingers in the desired direction, thereby moving thearbitrary virtual pedicle screw into the desired position and/ororientation in the patient's spine, e.g. centered in the target pedicle,towards the medial pedicle wall, towards the lateral pedicle wall,towards the superior pedicle wall and/or towards the inferior pediclewall, forward and, optionally, backward. As an alternative to virtuallymoving or aligning a virtual pedicle screw, a virtual predetermined pathfor a pedicle screw or for a vertebroplasty or kyphoplasty needle canalso be virtually moved or aligned, e.g. using a virtual interface orother interface.

The OHMD can display the virtual pedicle screw in any locationinitially, e.g. projected onto or outside the surgical field, e.g. alumbar, thoracic or cervical spine. The OHMD can optionally display thevirtual pedicle screw at a defined angle, e.g. orthogonal or parallel,relative to a fixed structure in the operating room, which can, forexample, be recognized using one or more cameras, image capture or videocapture systems integrated into the OHMD and spatial recognitionsoftware such as the one provided by Microsoft with the MicrosoftHololens or which can be recognized using one or more attached opticalmarkers or navigation markers including infrared or RF markers. Thevirtual pedicle screw can then displayed perpendicular or at anotherangle relative to the operating room table. The virtual pedicle screwcan be displayed at a defined angle to one or more anatomic orbiomechanical axes.

The surgeon can move the virtual pedicle screw to align it in thedesired location and/or orientation in the pedicle and/or vertebralbody. The surgeon can then evaluate the size of the virtual pediclescrew and the fit of the virtual pedicle screw by evaluating the sizeand fit of the virtual representation of the virtual pedicle screwsuperimposed onto the intended implantation site in the pedicle andvertebral body. The surgeon can move and align the virtual pediclescrew. If the virtual pedicle screw is too large for the patient'spedicle, the surgeon can cancel the virtual display of the particularsize of virtual pedicle screw displayed and the surgeon can select asmaller virtual pedicle screw from the library of virtual and physicalpedicle screws. If the virtual pedicle screw is too small for apatient's pedicle, the surgeon can cancel the virtual display of theparticular size of virtual pedicle screw displayed and the surgeon canselect a larger virtual pedicle screw from the library of virtual andphysical pedicle screws. In this manner, the surgeon can optimize thepedicle screw size and fit in three-dimensions in the actual surgicalsite, level by level.

Virtual Surgical Plans

Virtual and physical surgical instruments and implant components can beregistered in a common coordinate system, for example with one or moreOHMDs and live data of the patient. When pre-operative imaging studies,intra-operative imaging studies or intra-operative measurements areregistered in a common coordinate system with one or more OHMDs using,for example, anatomic features, anatomic landmarks, implantable andattachable markers, calibration and registration phantoms includingoptical markers, LEDs with image capture, navigation markers, infraredmarkers, RF markers, IMUs, or spatial anchors and spatial recognition,one or more of an instrument or implant position, orientation, alignmentcan be predetermined using the information from the pre- andintra-operative imaging studies and/or the intra-operative measurements.

In some embodiments of the invention, a surgeon or an operator candevelop a virtual surgical plan. The virtual surgical plan can includethe virtual removal of select tissues, e.g. bone or cartilage orsoft-tissue, e.g. for installing or implanting a medical device. Thevirtual surgical plan can include removal of a tumor or other tissues.The virtual surgical plan can include placing a graft or a transplant.Any surgical procedure known in the art can be simulated in a virtualsurgical plan, for example spinal fusion including anterior andposterior, spinal disk replacement using motion preservation approaches,hip replacement, knee replacement, ankle replacement, shoulderreplacement, ACL repair or reconstruction, ligament reconstruction.

A virtual surgical plan can be developed using intra-operative data ormeasurements, including measurements obtained using one or more opticalmarkers which can, for example, be detected using one or more cameras,an image capture system, a video capture system integrated into,attached to or separate from an OHMD. The one or more cameras, an imagecapture system, a video capture system integrated into, attached to orseparate from an OHMD can, for example, detect the coordinates of one ormore optical markers attached to the surgical site, e.g. a bone orcartilage, an altered surgical site, e.g. a bone cut, the operating roomtable, an extension of the operating room table, and/or fixturestructures in the operating room, e.g. walls. The one or more cameras,an image capture system, a video capture system integrated into,attached to or separate from an OHMD can detect the one or more opticalmarkers in static positions and/or dynamic, moving positions. Thecoordinates (x, y, z) of the optical markers can be measured in staticand dynamic conditions.

Any other sensor described in the specification, e.g. IMUs, navigationmarkers, e.g. infrared markers and/or RF markers, LEDs, can be used forobtaining intraoperative measurements and can be combined, for examplewith optical marker measurements, for deriving intra-operativemeasurements and for generating and/or developing a virtual surgicalplan.

Intra-operative measurements using one or more cameras, an image capturesystem, a video capture system integrated into or attached to an OHMDcan be beneficial when measurements are desired to be obtained from theview angle of the surgeon or, when multiple OHMDs are used, from theview angle of a surgical assistant or second surgeon. Intra-operativemeasurements using one or more cameras, an image capture system, a videocapture separate from an OHMD can be advantageous when measurements aredesired to be obtained from a view angle other than the surgeon or, whenmultiple OHMDs are used, from a view angle other than of a surgicalassistant or second surgeon.

Pre-operative data, e.g. pre-operative imaging studies or kinematicstudies of a patient, e.g. with the joint or the spine measured orimaged in motion, can also be incorporated into a virtual surgical plan.Pre-operative data alone can be used to develop a virtual surgical plan.

The virtual surgical plan can be developed with use of a computer orcomputer workstation as well as a local or remote computer or computernetwork. The computer or computer workstation can include one or moredisplays, keyboard, mouse, trackball, mousepad, joystick, human inputdevices, processor, graphics processors, memory chips, storage media,disks, and software, for example for 3D reconstruction, surfacedisplays, volume displays or CAD design and display, as well as optionalCAM output. The software can include one or more interfaces for CADdesign, for displaying the patient's anatomy, for displaying virtualsurgical instruments and for displaying virtual implants, implantcomponents, medical devices and/or medical device components.

The different anatomic and pathologic structures as well as thedifferent virtual instruments, e.g. virtual surgical guides includingdrill guides or cut blocks, virtual implants, implant components,medical devices and/or medical device components can optionally bedisplayed simultaneously on the same screen or screen section ornon-simultaneously, e.g. on different screens, on the same screen atdifferent times, or no different screen sections. The different anatomicand pathologic structures including hidden and/or obscured or partiallyhidden and/or obscured anatomic and pathologic structures as well as thedifferent virtual instruments, e.g. virtual surgical guides includingdrill guides or cut blocks, virtual implants, implant components,medical devices and/or medical device components can optionally bedisplayed using different colors or different shading. Some of thedifferent anatomic and pathologic structures as well as the differentvirtual instruments, e.g. virtual surgical guides including drill guidesor cut blocks, virtual implants, implant components, medical devicesand/or medical device components can optionally be displayed in a formof outline mode or pattern mode, where only the outline or selectfeatures or patterns of the anatomic and pathologic structures as wellas the virtual instruments, e.g. virtual surgical guides including drillguides or cut blocks, different virtual implants, implant components,medical devices and/or medical device components are being displayed,for example with solid, dotted or stippled lines or geometric patterns.

FIG. 11 shows how a virtual surgical plan 141 can be generated usingintraoperative data, e.g. intra-operative measurements 140, for examplemeasurements obtained with one or more cameras, an image capture systemor a video capture system integrated into, attached to or separate froman optical head mount display. Intraoperative measurements 140 can beutilized to generate a virtual surgical plan 141 which can be registeredin a common coordinate system 142. The intraoperative measurements 140can also be directly registered in the common coordinate system 142.Preoperative and/or intraoperative scan data 143 can be generated andcan be optionally displayed 144 in two or three dimensions in an OHMD145. Preoperative and/or intraoperative scan data 143 can optionally beincorporated 146 in the virtual surgical plan 141. Optical markers 147can be present on the patient, the surgical field, surgical instrumentsor implants and can be measured with regard to their position, location,orientation, direction of movement and/or speed 148. A virtual plane orpath or axis 149 can be displayed by the OHMD 145 and, using a virtualinterface 150, the plane or path or axis, as well as optionally virtualimplants or instruments, can be moved by the surgeon. Optionally, theOHMD 145 can display hidden or internal structures 151, e.g. visualizedon preoperative or intraoperative imaging studies or combinations ofboth, and the surgeon or the software can align the planes, axis orpath, as well as optionally virtual implants or instruments, relative tothe hidden or internal structures 149. The plane, axis or path orvirtual surgical instruments or virtual implants can be moved to betangent with or intersect anatomic landmarks, and/or anatomical axesand/or biomechanical axes 152, for example for alignment purposes or toachieve a predetermined position and/or orientation of an instrument oran implant. The OHMD can project stereoscopic views for the left eye andright eye by displaying electronic holograms with virtual datasuperimposing the virtual data using the left eye position andorientation on the live data for the left eye 153 and superimposing thevirtual data using the right eye position and orientation on the livedata for the right eye 154. The projected virtual data in 153 and 154can be used to position, orient, align, direct or place one or more of asurgical instrument, an implant component and an implant in relationshipto the live data of the patient, e.g. in a predetermined position,orientation, alignment direction or place 155. The position,orientation, alignment direction or place of the one or more of asurgical instrument, an implant component and an implant can optionallybe aligned with hidden anatomy or internal structures 151, optionallyusing a virtual interface 150. Someone skilled in the art can recognizethat multiple coordinate systems can be used instead of a commoncoordinate system. In this case, coordinate transfers can be appliedfrom one coordinate system to another coordinate system, for example forregistering the OHMD, live data of the patient including the surgicalsite, virtual instruments and/or virtual implants and physicalinstruments and physical implants.

FIG. 12 is another exemplary workflow for generating a virtual surgicalplan. Imaging data of a patient are acquired, e.g. at a site remote fromthe operating room 290. The imaging data can be transferred to acomputer or workstation, e.g. via electronic data transfer routines suchas ftp or internet 291. The imaging data of the patient can bereconstructed in three dimensions 292. The imaging data can be displayedin two or three dimensions on a computer display 293 or OHMD.

FIG. 13 shows an example how a virtual surgical plan 157 can be modifiedusing intraoperative data, e.g. intraoperative measurements 140. Thevirtual surgical plan 157 can be developed using pre-operative andintra-operative imaging data of the patient 143. The virtual surgicalplan 157 can be registered in a common coordinate system 142.Preoperative and/or intraoperative scan data 143 can be generated andcan be optionally displayed 144 in two or three dimensions in an OHMD145. Preoperative and/or intraoperative scan data 143 can be used todevelop the virtual surgical plan 157 which can be optionally displayed158 by the OHMD 145. Optical markers 147 can be present on the patient,the surgical field, surgical instruments or implants and can be measuredwith regard to their position, location, orientation, direction ofmovement and/or speed 148. A virtual plane or path or axis 149 can bedisplayed by the OHMD 145 and, using a virtual interface 150, the planeor path or axis, as well as optionally virtual implants or instruments,can be moved by the surgeon. Optionally, the OHMD 145 can display hiddenor internal structures 151, e.g. visualized on preoperative orintraoperative imaging studies or combinations of both, and the surgeoncan align the planes, axis or path, as well as optionally virtualimplants or instruments, relative to the hidden or internal structures149. The plane, axis or path or virtual surgical instruments or virtualimplants can be moved to be tangent with or intersect anatomiclandmarks, and/or anatomical axes and/or biomechanical axes 152, forexample for alignment purposes or to achieve a predetermined positionand/or orientation of an instrument or an implant. The OHMD can projectstereoscopic views for the left eye and right eye by displaying virtualdata superimposing the virtual data using the left eye position andorientation on the live data for the left eye 153 and superimposing thevirtual data using the right eye position and orientation on the livedata for the right eye 154. The projected virtual data in 153 and 154can be used to position, orient, align, direct or place one or more of asurgical instrument, an implant component and an implant in relationshipto the live data of the patient, e.g. in a predetermined position,orientation, alignment direction or place 155. The position,orientation, alignment direction or place of the one or more of asurgical instrument, an implant component and an implant can optionallybe aligned with hidden anatomy or internal structures 151, optionallyusing a virtual interface 150. Intraoperative measurements 140 can beutilized to generate or modify a virtual surgical plan 157. The virtualsurgical plan 157 and/or a modified virtual surgical plan 162 canoptionally be superimposed on preoperative and intraoperative imagingdata of the patient 159. The virtual surgical plan 157 and/or a modifiedvirtual surgical plan 162 can optionally be superimposed on preoperativeand intraoperative imaging data of the patient 159. The modified virtualsurgical plan 162 can be further modified based on visual or opticalfeedback or input 161 and it can be used to position, orient, align,direct, place one or more virtual or physical instruments, implantcomponents and/or implants in a predetermined position 155. Someoneskilled in the art can recognize that multiple coordinate systems can beused instead of a common coordinate system. In this case, coordinatetransfers can be applied from one coordinate system to anothercoordinate system, for example for registering the OHMD, live data ofthe patient including the surgical site, virtual instruments and/orvirtual implants and physical instruments and physical implants.

In some embodiments of the invention, one or more of a virtual surgicaltool, virtual surgical instrument including a virtual surgical guide orcut block, virtual trial implant, virtual implant component, virtualimplant or virtual device, one or more of a predetermined start point,predetermined start position, predetermined start orientation oralignment, predetermined intermediate point(s), predeterminedintermediate position(s), predetermined intermediate orientation oralignment, predetermined end point, predetermined end position,predetermined end orientation or alignment, predetermined path,predetermined plane, predetermined cut plane, predetermined contour oroutline or cross-section or surface features or shape or projection,predetermined depth marker or depth gauge, predetermined angle ororientation or rotation marker, predetermined axis, e.g. rotation axis,flexion axis, extension axis, predetermined axis of the virtual surgicaltool, virtual surgical instrument including virtual surgical guide orcut block, virtual trial implant, virtual implant component, implant ordevice, non-visualized portions for one or more devices or implants orimplant components or surgical instruments or surgical tools, and/or oneor more of a predetermined tissue change or alteration can be moved,re-oriented and/or re-aligned by the surgeon using a virtual or otherinterface. For example, the virtual representation of the one or more ofa virtual surgical tool, virtual surgical instrument including a virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, virtual implant or virtual device, one or more of apredetermined start point, predetermined start position, predeterminedstart orientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration can include a “touch area”, wherein an image or video capturesystem and gesture recognition software, for example the one provided byMicrosoft with the Microsoft Hololens including, for example, theintegrated virtual “drag function” for holograms can be used to move thevirtual data. For example, one or more cameras integrated or attached tothe OHMD can capture the movement of the surgeon's finger(s) inrelationship to the touch area; using gesture tracking software, thehologram(s) can then be moved by advancing the finger towards the toucharea in a desired direction. A surgeon can, for example, also “hold” thehologram(s) by closing two fingers, e.g. thumb and index finger, overthe touch area and then moving the fingers in the desired direction.

Placement Rules, Selection Rules, Design Rules

A virtual surgical plan can optionally include placement rules forsurgical instruments and/or medical devices, implants or implantcomponents. These placement rules can be based on standard rules ofsurgery or on standard surgical techniques, e.g. placement rules of kneearthroplasty, hip arthroplasty or for pedicle screws. Placement rules orselection rules or design rules for a virtual surgical plan can be basedon the patient's anatomy, desired implant, component or medical deviceposition, location, orientation, rotation or alignment, one or moreanatomical axes, one or more biomechanical axes, a mechanical axis ofthe knee or lower extremity, one or more rotational axes, a desiredfunction of an implant, implant component or medical device. Placementrules or selection rules or design rules for a surgical plan can beused, for example, to select an implant. Placement rules or selectionrules or design rules can include implant, implant component, or medicaldevice dimensions or shape. Placement rules or selection rules or designrules can include avoidance of certain soft-tissues, vessels or neuralstructures as well as other sensitive tissues or structures, e.g.ligaments intended to be preserved. For example, in unicompartmentalarthroplasty, a placement rule can include that a vertical tibial cutspares the medial tibial spine. In cruciate retaining total kneearthroplasty, a placement rule can include to spare the posteriorcruciate ligament during the tibial resection, for example by designinga bone cut in a manner to avoid the posterior cruciate ligament.Placement rules, selection rules or design rules of a virtual surgicalplan can include demographic information of the patient, e.g. weight,height, age, gender, other information such as bone mineral density orstructure, clinical history, history of prior fractures, or functionalinformation, e.g. on motion of a joint, or metabolic information, e.g.for certain organs or pathologic tissues. Automatic placement of avirtual medical device, device component or implant is possible, forexample based on anatomic criteria, pathologic criteria, or functionalcriteria using placement rules, selection rules or design rules forvirtual surgical plans. Placement of a virtual medical device usingplacement rules, selection rules or design rules can be manual,semi-automatic or automatic. Manual, semi-automatic or automaticplacement rules will typically require a software and a user interface.

For example, in spinal surgery the placement of a pedicle screw in thevirtual surgical plan can be based on

-   -   Distance between pedicle screw or related bone void to accept        the pedicle screw to the medial, lateral, superior, and/or        inferior endosteal surface or cortical surface in portions or        all of the pedicle.    -   Area or volume between pedicle screw or related bone void to        accept the pedicle screw to the medial, lateral, superior,        and/or inferior endosteal surface or cortical surface in        portions or all of the pedicle.

The foregoing information on distance or area can also be used forselecting a size, width, diameter or length of a pedicle screw.

In spinal surgery the placement of a pedicle screw in the virtualsurgical plan can also be based on:

-   -   Location of the pedicle screw including its tip in the vertebral        body.    -   Location of the pedicle screw including its tip in relationship        to a spinal/vertebral body fracture.    -   Location of the pedicle screw including its tip in relationship        to a superior endplate.    -   Location of the pedicle screw including its tip in relationship        to an inferior endplate.    -   Location of the pedicle screw including its tip in relationship        to an anterior vertebral cortex.    -   Location of the pedicle screw including its tip in relationship        to a vessel.    -   Location of the pedicle screw including its tip in relationship        to the aorta.    -   Location of the pedicle screw including its tip in relationship        to the inferior vena cava.    -   Location of the pedicle screw including its tip in relationship        to neural structures, the thecal sac, nerve roots and/or the        spinal cord.    -   Distance, area or volume between the pedicle screw including its        tip to a spinal/vertebral body fracture.    -   Distance, area or volume between the pedicle screw including its        tip to a superior endplate.    -   Distance, area or volume between of the pedicle screw including        its tip to an inferior endplate.    -   Distance, area or volume between the pedicle screw including its        tip to an anterior vertebral cortex.    -   Distance, area or volume between the pedicle screw including its        tip to a vessel.    -   Distance, area or volume between the pedicle screw including its        tip to the aorta.    -   Distance, area or volume between the pedicle screw including its        tip to the inferior vena cava.    -   Distance, area or volume between the pedicle screw including its        tip to neural structures, the thecal sac, nerve roots and/or the        spinal cord.

The foregoing information on location or distance or area or volume canalso be used for selecting a size, width, diameter or length of apedicle screw.

The placement and the selection of a pedicle screw in spinal surgery canbe based on any of the foregoing including any combinations thereof.

The surgeon can receive 2D or 3D or multi-dimensional information of thepatient. The information can be displayed, for example using a displayscreen, e.g. a computer screen separate from the OHMD or the OHMD. Thesurgeon can mark anatomic structures or pathologic structures on thecomputer screen using the 2D or 3D or multi-dimensional information ofthe patient. The information can optionally be segmented or can bemodified, for example using image processing techniques known in theart. The marking can be performed using the display of the OHMD unit,e.g. using a virtual user interface.

The surgeon can also mark sensitive tissue, e.g. nerves, brainstructure, vessels etc., that the surgeon wants to preserve or protectduring the surgery. Such sensitive structure(s) can be highlighted, forexample using different colors, when the virtual surgical plan and therelated anatomic data or pathologic tissue information is beingtransmitted to or displayed by the OHMD. The surgical plan can bedesigned, adapted or modified so that sensitive structures are avoidedor only minimally perturbed. For example, if a virtual surgical planwould result in an interference between a surgical instrument, e.g. ascalpel, a saw, a drill or a bur and a sensitive structure such as avessel or a nerve, the virtual surgical plan can be adapted or modifiedby moving the position, location, orientation and/or direction of thevirtual surgical instrument in order to avoid any interference orcontact of the sensitive structure(s) with the surgical instrument. Themarking can be performed using the display of the OHMD unit, e.g. usinga virtual user interface. For example, the surgeon can optionally pointat or circle with his or her finger sensitive structure on the livesurgical site including by optionally touching the sensitive tissue. Oneor more cameras, an image or video capture system integrated into,attached to or separate from the OHMD can detect the finger movement andcan highlight the sensitive areas pointed out or circled by thesurgeon's finger.

In some embodiments of the invention, if an interference or contactbetween the surgical instrument and one or more sensitive structurescannot be avoided (in the virtual data and/or the live or physicalsurgery), the virtual surgical plan can be adapted or modified to move,typically at least partially, the sensitive structure(s), for exampleusing tissue retractors, in order to minimize or reduce any interferenceor contact of the surgical instrument with the sensitive structure(s).

In some embodiments of the invention, if an interference or contactbetween the surgical instrument and one or more sensitive structurescannot be avoided (in the virtual data and/or the live or physicalsurgery), the virtual surgical plan can be adapted or modified toprotect, at least partially, the sensitive structure(s), for exampleusing a virtual and in the live patient physical metal or plastic shieldwhich can optionally be interposed between the sensitive structure(s)and the surgical instrument in order to minimize or reduce anyinterference or contact of the surgical instrument with the sensitivestructure(s).

The surgeon can mark the desired location, position, orientation, and oralignment of a graft, transplant or an implant or components thereof.Implant materials can include organic and inorganic matter. Implantmaterials can include biologic and non-biologic matter.

In a hip replacement procedure, for example, the surgeon can indicatethe desired location, position, orientation, alignment, anteversion oroffset of an acetabular component or a femoral component. With thefemoral component, the surgeon can also indicate the desired femoralneck resection level and the desired position of the component in thefemoral canal including the desired entry point into the cut femoralneck, e.g. medially, laterally, anteriorly or posteriorly as well as thedesired entry angle. With the acetabular component, the surgeon can alsoindicate the desired reaming depth and any desired medialization orlateralization.

With the implantation of any medical device, the surgeon can indicatethe desired location, position, orientation, alignment of the medicaldevice. Thus, the virtual surgical plan can show the desired location,position, orientation, or alignment of a medical device. The virtualsurgical plan can also show the desired location, position, orientation,or alignment of a medical device relative to neighboring tissue.Neighboring tissue can be the tissue of the same organ or joint.Neighboring tissue can also be the tissue of adjacent sensitivestructures, e.g. vessel, nerves, other organs and the like.

The surgeon can optionally simulate different locations, positions,orientations or alignments of a medical device. The simulation ofdifferent locations, positions, orientations or alignments of a medicaldevice can be particularly helpful when the medical device entails morethan one component as can be the case, for example, with

-   -   Pedicle screws, connectors and spinal rods    -   Artificial intervertebral disks, e.g. metallic endplates and        ultra-high molecular weight polyethylene mobile sliding core    -   Knee replacement components, including tibial tray, polyethylene        inserts, femoral components, mobile bearings    -   Hip replacement components, including acetabular cup, acetabular        liner, femoral head, optionally modular femoral neck, femoral        stem or mono-block femoral neck and stem

With these multicomponent devices, the surgeon can plan the placement ofindividual components in the virtual surgical plan and the surgeon canoptionally evaluate their location, position, orientation or alignmentrelative to each other. The surgeon can then make adjustments to theplacement, e.g. the position, location, orientation, rotation oralignment of one or more of the components in the virtual plan and,optionally later, in the live surgery. Optionally, the surgeon can alsotest the function of these components in relationship to each other. Forexample, in a surgical plan for an artificial intervertebral disk, thesoftware can allow the surgeon to virtually simulate spinal flexion orextension or lateral bending to the left and right with one or more ofthe medical device components included in the virtual surgical plan orthe motion simulation. The surgeon can repeat the virtual surgical planor the simulation with different degrees of flexion or extension orlateral bending to the left and the right and/or with differently sizedor shaped medical devices or medical device components. If there isinterchangeability of parts or components between different sizes andshapes or a medical device, the surgeon can optionally repeat thevirtual surgical plan or the simulation using such different sizecomponents, e.g. a large size polyethylene insert or spacer with amedium size metal backing components or vice versa.

The surgeon can optionally superimpose medical device components withdifferent size and/or shapes on the information and select the devicecomponent(s) that best fit the patient or that best match the patient.

In some embodiments of the invention, when, for example, a virtualsurgical plan is developed using pre-operative data, e.g. pre-operativeimaging data, the information is sent from the surgeon's or operator'soffice, e.g. a radiology office, to a central site, e.g. for imageprocessing or for generating an initial draft surgical plan resulting inprocessed data or information. The processed information can betransmitted back to the surgeon or the operator. The surgeon or theoperator can review the draft surgical plan. The surgeon or the operatorcan accept the draft surgical plan. The surgeon or the operator canoptionally modify the draft surgical plan. The accepted or modifieddraft surgical plan can optionally be transmitted back to the centralsite. The central site can, for example, generate instructions to shipcertain medical device components that the surgeon has accepted orselected with the accepted or modified surgical plan.

When intra-operative data are used for developing the virtual surgicalplan, the surgeon can develop portions or the entire virtual surgicalplan on his or her own, for example using a computer, standard hardwarecomponents, display and software in his or her office, a computer,standard hardware components, display and software in the operatingroom, or the optical head mount display, e.g. using a virtual interface,or combinations thereof. Different computers including the OHMD can beconnected via a network, e.g. a WiFi or LiFi network.

The surgeon can optionally incorporate pre-operative data into thevirtual surgical plan. For example, in knee replacement, the surgeon canperform intra-operative measurements using, for example, optical markersto determine the mechanical axis of the leg and to define femoral and/ortibial and/or patellar landmarks and register them in the commoncoordinate system and can be used for the virtual surgical plan whichcan also be registered in the common coordinate system. The surgeon canthen incorporate or import data from one or more pre-operative and/orintra-operative knee x-rays, for example femoral, tibial or patellarcomponent size and/or desired varus or valgus correction and/or desiredfemoral and/or tibial component rotation and/or desired femoralcomponent flexion and/or desired tibial slope, and/or desired femoral,tibial and/or patellar component position and/or orientation and/oralignment into the virtual surgical plan. Standard data, e.g. a fixedtibial slope, e.g. 0 degrees, 3 degrees or 5 degrees can also beincorporated into the virtual surgical plan. Any of the foregoing can beregistered in the common coordinate system and optionally virtuallydisplayed by the OHMD.

In hip replacement, the surgeon can perform intra-operative measurementsusing, for example, optical markers to determine the location of thecenter of rotation of the hip joint, to define femoral and acetabularlandmarks, e.g. the top of the greater trochanter, the sulcus point,e.g. the lowest point between the greater trochanter and the femoralneck, and the lesser trochanter, the acetabular rim and/or the center ofthe acetabulum, e.g. by pointing at them using a pointer with one ormore attached optical markers; these and other intra-operativemeasurements can be registered in the common coordinate system and canbe used for the virtual surgical plan which can also be registered inthe common coordinate system. The surgeon can then incorporate or importdata from one or more pre-operative and/or intra-operative hip x-raysand/or pelvic x-rays, for example femoral and acetabular component size,desired liner including lipped and offset liners, desired femoral headsize including plus and minus head sizes, and/or desired leg length,and/or desired center of rotation, and/or desired femoral neck length,and/or desired femoral neck angle, and/or desired femoral and/oracetabular component anteversion and/or offset, including combinedanteversion. Standard data, e.g. a fixed femoral, acetabular or combinedanteversion, a fixed femoral neck angle, a range of angles for anacetabular safe zone can also be incorporated into the virtual surgicalplan. Any of the foregoing can be registered in the common coordinatesystem and optionally virtually displayed by the OHMD.

In some embodiments of the invention, aspects of the surgical plan, e.g.the intended location of a medical device that the surgeon is planningto implant can be displayed by the OHMD superimposed onto the live data.The intended location can be indicated, for example, by a virtualmedical device component that is a representation of the medical devicecomponent selected for implantation. The virtual medical devicecomponent displayed by the OHMD in superimposition with the live datacan be displayed, for example, in its final desired position. Thesurgeon can then intraoperatively place or insert the medical devicecomponent aligning the physical device with the virtual devicecomponent.

In some embodiments, the intended location of a graft, transplant,medical device or other implantable can be indicated using virtualmarkers or targets displayed by the OHMD simultaneous with the live dataof the patient. The surgeon can then align the graft, transplant,medical device or other implantable with the virtual markers or targetsor the surgeon can direct the graft, transplant, medical device or otherimplantable towards the virtual markers or targets.

A visual or acoustic or other warning signal can be emitted or providedif the surgeon/operator deviates from the surgical plan. The visualwarning signal can be provided by the OHMD, e.g. a red backgroundflashing in the display of the virtual data or a color change, e.g. tored, of the virtual data.

In some embodiments of the invention, the virtual surgical plan canstart by selecting or designing a desired implant or implant componentor medical device size and/or dimension and/or shape based on thepatient's anatomy, surgical site, pathologic conditions, deformity andother information including but not limited to a desired location,position, orientation, rotation or alignment in relationship to one ormore anatomic or rotational or biomechanical axes. The selection ordesign of the desired size and/or dimension and/or shape can be followedby the placement of the implant, implant component or medical device inthe desired location, position, orientation, rotation or alignment inrelationship to one or more anatomic or biomechanical axes, thepatient's anatomy surgical site, pathologic conditions or deformity. Theprocess can be iterative. For example, the implant or implant componentor medical device selection or design can be followed by a desiredplacement, which can be followed by changes in the selection or designof the implant or implant component or medical device selection, whichcan be followed by adjustments in placement and so forth. The iterativeprocess can be automatic or semiautomatic.

Once the final implant selection or design and placement have beendetermined in the virtual surgical plan, the preceding surgical stepscan be designed or selected in the virtual surgical plan in relationshipto the patient's anatomy, the surgical site, the pathologic condition,one or more anatomic or biomechanical axes, functional information,information on sensitive tissues and other tissues. The precedingsurgical steps can be designed or selected in reverse order startingwith the final implant or implant component or medical device placement,in consecutive order or in random order or any combinations thereof.Surgical steps can be optionally repeated to optimize any tissuealterations and/or implant placement and/or implant selection and/orimplant design. If a virtual surgical plan indicates the potential forcomplications during the surgery, e.g. placement too close to a vesselor neural structure or other sensitive structure, the surgical plan,portions of the surgical plan, the sequence of the surgical plan and theimplant, implant component or medical device selection or design can bemodified in order to avoid such potential complications. Thus, theentire process between selection and placement of the implant andsurgical steps including display of surgical instruments can beiterative in the virtual surgical plan.

In some embodiments of the invention, the virtual surgical plan canstart by placing a virtual implant or implant component or medicaldevice in a desired location, position, orientation, rotation oralignment in relationship to one or more anatomic or biomechanical axes,the patient's anatomy surgical site, pathologic conditions or deformity.The implant used for this initial or final placement can be an implantselected from an average, a minimum or a maximum size, dimension orshape or combinations thereof. The placing of the implant or implantcomponent or medical device can then be followed by the selection ordesign of a desired implant or implant component or medical device sizeand/or dimension and/or shape. The process can be iterative. Forexample, placement of the implant or implant component or medical devicecan be followed by a selection or design of the desired the implant orimplant component or medical device size, dimension or shape, which canbe followed by changes in the placement of the implant or implantcomponent or medical device, which can be followed by changes in theselection or design of size, dimension or shape and so forth. Theiterative process can be automatic or semiautomatic.

Once the final implant placement and selection or design have beendetermined in the virtual surgical plan, the preceding surgical stepscan be designed or selected in the virtual surgical plan in relationshipto the patient's anatomy, the surgical site, the pathologic condition,one or more anatomic or biomechanical axes, functional information,information on sensitive tissues and other tissues. The precedingsurgical steps can be designed or selected in reverse order startingwith the final implant or implant component or medical device placement,in consecutive order or in random order or any combinations thereof.Surgical steps can be optionally repeated to optimize any tissuealterations and/or implant placement and/or implant selection and/orimplant design. If a virtual surgical plan indicates the potential forcomplications during the surgery, e.g. placement too close to a vesselor neural structure or other sensitive structure, the surgical plan,portions of the surgical plan, the sequence of the surgical plan and theimplant, implant component or medical device selection or design can bemodified in order to avoid such potential complications. Thus, theentire process between selection and placement of the implant andsurgical steps including display of surgical instruments can beiterative in the virtual surgical plan.

In some embodiments of the invention, the virtual surgical plan canstart out with the initial surgical step as defined, for example, in thesurgical technique. This can be followed optionally by each or some ofthe subsequent surgical steps, for example only the major steps. Thevirtual surgical plan can then continue up to the selection and/ordesign and placement of the implant in the virtual data of the patient.If the resultant selection and/or design and/or placement of theimplant, implant component or medical device differs from the desiredresult, for example as defined in the surgical plan or as desired by thesurgeon, any of the foregoing surgical steps, the placement and/or theselection or the design of the implant, implant component or medicaldevice can be modified. This process can be iterative, manual,semi-automatic or automatic until the desired virtual surgical plan,implant, implant component or medical device selection and/or design orplacement are achieved.

FIG. 14 shows an illustrative example how multiple OHMDs can be usedduring a surgery, for example by a first surgeon, a second surgeon, asurgical assistant and/or one or more nurses and how a surgical plan canbe modified and displayed during the procedure by multiple OHMDs whilepreserving the correct perspective view of virtual data andcorresponding live data for each individual operator. A system 10 forusing multiple OHMDs 11, 12, 13, 14 for multiple viewer's, e.g. aprimary surgeon, second surgeon, surgical assistant(s) and/or nurses(s)is shown. The multiple OHMDs can be registered in a common coordinatesystem 15 using anatomic structures, anatomic landmarks, calibrationphantoms, reference phantoms, optical markers, navigation markers,and/or spatial anchors, for example like the spatial anchors used by theMicrosoft Hololens. Pre-operative data 16 of the patient can also beregistered in the common coordinate system 15. Live data 18 of thepatient, for example from the surgical site, e.g. a spine, optionallywith minimally invasive access, a hip arthrotomy site, a knee arthrotomysite, a bone cut, an altered surface can be measured, for example usingone or more IMUs, optical markers, navigation markers, image or videocapture systems and/or spatial anchors. The live data 18 of the patientcan be registered in the common coordinate system 15. Intra-operativeimaging studies 20 can be registered in the common coordinate system 15.OR references, e.g. an OR table or room fixtures can be registered inthe common coordinate system 15 using, for example, optical markersIMUs, navigation markers or spatial mapping 22. The pre-operative data16 or live data 18 including intra-operative measurements orcombinations thereof can be used to develop, generate or modify avirtual surgical plan 24. The virtual surgical plan 24 can be registeredin the common coordinate system 15. The OHMDs 11, 12, 13, 14 canmaintain alignment and superimposition of virtual data of the patientand live data of the patient for each OHMD 11, 12, 13, 14 for eachviewer's perspective view and position and head position and orientation27. Using a virtual or other interface, the surgeon wearing OHMD 1 11can execute commands 32, e.g. to display the next predetermined bonecut, e.g. from a virtual surgical plan or an imaging study orintra-operative measurements, which can trigger the OHMDs 11, 12, 13, 14to project virtual data of the next surgical step 34 superimposed ontoand aligned with the surgical site in a predetermined position and/ororientation. Any of the OHMDs 11, 12, 13, 14 can acquire one or moreoptical measurements or measurement inputs, e.g. of anatomic landmarks,axes from cameras, anatomic axes, biomechanical axes, a mechanical axisof a leg 17, using for example an integrated or attached camera, imagecapture or video system. By using multiple OHMDs 11, 12, 13, 14 fromdifferent view angles with multiple cameras, image capture or videosystems, the accuracy of the measurements can optionally be improved.Optionally, parallax measurements can be performed using the multipleOHMDs 11, 12, 13, 14 from different view angles with multiple cameras,image capture or video systems. The one or more optical measurements canbe used to modify the virtual surgical plan 19, optionally using theinformation from multiple OHMDs 11, 12, 13, 14. Someone skilled in theart can recognize that multiple coordinate systems can be used insteadof a common coordinate system. In this case, coordinate transfers can beapplied from one coordinate system to another coordinate system, forexample for registering the OHMD, live data of the patient including thesurgical site, virtual instruments and/or virtual implants and physicalinstruments and physical implants.

Tissue Morphing Including Bone Morphing, Cartilage Morphing

In some embodiments of the invention, the shape of one or more of thepatient's tissues, such as a bone, a cartilage, a joint or an organ, canbe estimated or morphed in three dimensions intra-operatively, e.g.during the surgery. The estimating or morphing of the patient's tissueshape, e.g. bone shape, cartilage shape, joint shape or organ shape, canhelp reduce or obviate the need for pre-operative imaging and, in selectembodiments, intra-operative imaging.

In some embodiments of the invention, 2D preoperative data can be usedand the shape of one or more of the patient's tissues, such as a bone, acartilage, a joint or an organ, can be estimated or morphed in threedimensions pre-operatively, e.g. prior to surgery.

Bone Morphing and/or Cartilage and/or Tissue Morphing UsingPre-Operative Imaging or Intra-Operative Imaging

In some embodiments of the invention, one or more two-dimensional imagesof the patient can be obtained. These images can, for example, includeone or more x-rays of the patient. X-rays can be obtained using digitalacquisition techniques. X-rays can also be obtained using conventionalfilm based technique, in which case the x-rays can be subsequentlydigitized using a scanner.

Exemplary x-ray images can include:

-   -   Spine: AP, PA, lateral, oblique views, and/or angled views,        flexion, extension views, lateral bending views; upright, supine        or prone    -   Hip: AP, PA, lateral, oblique views, angled views, and/or        frogleg view; standing or lying, weight-bearing or        non-weight-bearing    -   Knee: AP, PA, lateral, oblique views, angled views, and/or        Merchant view, sunrise view and the like; standing or lying,        weight-bearing or non-weight-bearing    -   Full leg x-rays films; standing or lying, weight-bearing or        non-weight-bearing    -   Selective leg x-rays films, e.g. hip, knee, ankle; standing or        lying, weight-bearing or non-weight-bearing

X-rays can be obtained with the patient in upright, supine and/or proneposition. X-rays can be obtained with the patient in weight-bearing andin non-weight-bearing position. In some embodiments of the invention,x-rays are obtained intra-operatively, for example with the patientalready positioned and placed for the intended surgical procedure.

The x-ray data of the patient can be transferred into a computer.Optionally, image processing can be applied to segment select patienttissues, such as a bone or vertebra or vertebral structure, subchondralbone, cortical bone, osteophytes. Image processing can, for example,also be applied to determine the edge of select patient tissues, such asa bone or vertebra or vertebral structure, subchondral bone, corticalbone, osteophytes. When subchondral bone has been identified and/orderived from the images, including a subchondral bone curvature and/orgeometry and/or shape, a cartilage shape, curvature or geometry can besuperimposed or added to the subchondral bone shape. The cartilageshape, curvature or geometry can assume a standard cartilage thicknessfor a given joint and/or a given patient, e.g. 1.0 mm, 1.5 mm, 2.0 mm,2.5 mm, 3.0 mm, 3.5 mm. The cartilage geometry can also assume avariable cartilage thickness, e.g. depending on the location of thecartilage in the joint and/or on the articular surface and/or based onthe patient's age, gender, race, body weight, and/or BMI, as well asunderlying deformity, e.g. varus or valgus deformity.

In some embodiments of the invention, the 2D x-rays images can be usedto derive information about the dimensions and shape of the anatomicstructure(s) included in the x-ray. Some of this information can be, forexample:

-   -   Anatomic landmark(s)    -   Distances and/or dimensions between two or more known        landmarks/structures    -   Angles between landmarks    -   Anatomic axes    -   Biomechanical axes    -   Curvature information    -   Surface information    -   Edge information    -   Curvature information    -   Shape information, e.g. when information from multiple x-rays        images obtained with different projection or beam angles is        combined or aggregated    -   Length information, e.g. in AP, ML, SI direction, AP, ML, SI        plane, oblique planes    -   Width information, e.g. in AP, ML, SI direction, AP, ML, SI        plane, oblique planes    -   Depth information, e.g. in AP, ML, SI direction, AP, ML, SI        plane, oblique planes

Examples of landmarks, distances, dimensions, surfaces, edges, angles,axes, curvatures, shapes, lengths, widths, depths and/or other featuresfor the spine, the hip, the knee and the shoulder joint that can be usedfor bone morphing and 3D model selection, development, derivations, anddeformations in any surgeries of these or to these areas are providedbelow. These examples are in no way meant to be limiting of theinvention, but are only exemplary in nature. Someone skilled in the artwill readily recognize other landmarks, distances, dimensions, surfaces,edges, angles, axes, curvatures, shapes, lengths, widths, depths and/orother features for these joints as well as any other joint in the humanbody.

Spine

-   -   Cortical bone of a pedicle    -   Endosteal bone of a pedicle    -   Posterior cortical bone of a vertebral body    -   Anterior cortical bone of a vertebral body    -   Lateral cortical bone of a vertebral body    -   Superior endplate    -   Inferior endplate    -   Intervertebral disk    -   Vertebral body    -   Trabecular bone of the vertebral body    -   Superior facet    -   Inferior facet    -   Spinous process    -   Any fracture components or fragments, e.g. involving a pedicle,        a facet joint or a vertebral body    -   Endplate shape, e.g. sagittal plane    -   Endplate shape, e.g. coronal plane    -   Schmorl's node(s)    -   Interpedicular distance    -   Intervertebral height or disk height    -   AP length of vertebral body, e.g. at level of inferior endplate,        superior endplate, mid-portion    -   ML width of vertebral body, e.g. at level of inferior endplate,        superior endplate, mid-portion    -   Oblique width vertebral body, e.g. at level of inferior        endplate, superior endplate, mid-portion    -   Vertebral body height, e.g. anterior, mid-portion, posterior    -   Pedicle length    -   Pedicle width    -   Pedicle height    -   Pedicle angle    -   Spinous process SI thickness, e.g. anterior, mid-portion, tip    -   Spinous process width, e.g. anterior, mid-portion, tip    -   Spinous process inferior angle from origin    -   Facet dimensions, AP, ML, SI    -   Facet angle, e.g. angle of joint formed between inferior facet        of superior vertebra and superior facet of inferior vertebra    -   Lamina SI height    -   Lamina AP width    -   Lamina ML radius, diameter    -   Spinal canal AP diameter, ML diameter    -   Lordosis    -   Kyphosis    -   Scoliosis    -   Side bending, e.g. left lateral, right lateral    -   Cobb angle    -   Lumbosacral angle

Hip

-   -   Lateral acetabular point or edge    -   Medial acetabular point or edge    -   Superior acetabular point or edge    -   Anterior acetabular point or edge    -   Posterior acetabular point or edge    -   Triradiate cartilage and region    -   Acetabular labrum, medial, lateral, anterior, posterior (e.g.        when x-ray contrast has been injected into the joint)    -   Fovea capitis    -   Femoral head subchondral bone, contour, outline    -   Femoral head neck/junction, curvature, convex, concave    -   Greater trochanter, e.g. lateral cortex, superior cortex,        anterior cortex, posterior cortex    -   Sulcus point (lowest point between greater trochanter and        femoral neck)    -   Sulcus curvature    -   Greater trochanter/sulcus transition, curvature, convex, concave    -   Lesser trochanter    -   Lesser trochanter/femoral neck transition, curvature    -   Lesser trochanter/femoral shaft transition    -   Femoral shaft, anterior cortex, posterior cortex, medial cortex,        lateral cortex    -   Anterior cortex, posterior cortex, medial cortex, lateral cortex        for any of the foregoing structures as applicable    -   Endosteal bone, anterior, posterior, medial, lateral for any of        the foregoing structures as applicable    -   Femoral neck angle    -   Femoral shaft angle    -   Acetabular angle    -   Acetabular anteversion    -   Femoral anteversion    -   Femoral shaft angle    -   Pelvic tilt    -   Femoral offset    -   Shenton's line    -   Hilgenreiner line    -   Perkin line    -   Acetabular index

Knee

-   -   Medial wall of the femoral notch    -   Lateral wall of the femoral notch    -   Roof of the femoral notch    -   Medial wall of the medial condyle    -   Lateral wall of the lateral condyle    -   Medial epicondylar eminence    -   Lateral epicondylar eminence    -   Medial femoral condyle shape, e.g. radii, convexities,        concavities    -   Lateral femoral condyle shape, e.g. radii, convexities,        concavities    -   Intercondylar notch shape    -   Intercondylar notch surface features    -   Medial tibial spine    -   Lateral tibial spine    -   Anteromedial tibial rim    -   Anterolateral tibial rim    -   Medial tibial rim    -   Lateral tibial rim    -   Lowest point of the medial plateau    -   Lowest point of the lateral plateau    -   Highest point of the medial plateau    -   Highest point of the lateral plateau    -   Medial tibial plateau shape    -   Lateral tibial plateau shape    -   Medial tibial plateau sagittal curvature    -   Lateral tibial plateau sagittal curvature    -   Medial tibial plateau coronal curvature    -   Lateral tibial plateau coronal curvature    -   Medial tibial plateau surface features, e.g. radii, convexities,        concavities    -   Lateral tibial plateau surface features, e.g. radii,        convexities, concavities    -   Femoral osteophytes    -   Tibial osteophytes    -   Patellar osteophytes    -   Femoral subchondral cysts    -   Tibial subchondral cysts    -   Patellar osteophytes    -   Patellar subchondral cysts    -   Trochlea osteophytes    -   Trochlea subchondral cysts    -   Patellar sagittal curvature    -   Patellar coronal curvature    -   Patellar axial curvature    -   Patellar surface features, e.g. radii, convexities, concavities    -   Patellar surface features, e.g. radii, convexities, concavities    -   Patellar circumference shape    -   Patellar rise    -   Patellar thickness    -   Trochlear depth    -   Trochlear sagittal curvature    -   Trochlear axial curvature    -   Trochlear coronal curvature    -   Trochlea sagittal shape    -   Trochlea axial shape    -   Trochlea coronal shape    -   Trochlear angle    -   Epicondylar axis    -   Posterior femoral axis    -   Trochlear rotation axis    -   Mechanical axis    -   Q-angle

Shoulder

-   -   Clavicle    -   AC joint    -   Acromion    -   Glenoid    -   Scapula    -   Coracoid    -   Humeral head    -   Humeral neck    -   Humeral shaft    -   Glenoid osteophytes    -   Humeral osteophytes    -   AC joint osteophytes    -   Glenoid subchondral cysts    -   Humeral subchondral cysts    -   AC joint subchondral cysts    -   Acromio-humeral distance    -   Acromio-humeral space    -   Deepest point of glenoid    -   Most anterior point or edge of glenoid    -   Most posterior point or edge of glenoid    -   Most superior point or edge of glenoid    -   Most inferior point or edge of glenoid    -   Glenoid shape    -   Humeral head shape    -   Glenoid sagittal curvature, e.g. radii, convexities, concavities    -   Glenoid axial curvature, e.g. radii, convexities, concavities    -   Glenoid coronal curvature, e.g. radii, convexities, concavities    -   Humeral head sagittal curvature, e.g. radii, convexities,        concavities    -   Humeral head axial curvature, e.g. radii, convexities,        concavities    -   Humeral head coronal curvature, e.g. radii, convexities,        concavities    -   Mechanical axis    -   Anatomical axis    -   Angle of inclination    -   Axis of head and neck    -   Axis through epicondyles    -   Angle of retroversion

By measuring any of the foregoing landmarks, distances, dimensions,surfaces, edges, angles, axes, curvatures, shapes, lengths, widths,depths and/or other features, it is possible to estimate a 3D shape,volume or surface(s) of a bone, e.g. a proximal femur, a distal femur, aproximal tibia, an acetabulum, a vertebral body and spinal elements anda glenoid and/or a proximal humerus. The more landmarks, distances,dimensions, surfaces, edges, angles, axes, curvatures, shapes, lengths,widths, depths and/or other features are being measured, the moreaccurate can the estimation of the 3D shape, volume or surface(s) of thebone be. In addition, the more 2D images are being taken or acquiredfrom different view angles, projection angles, beam angles, optionallywith the same magnification or different magnifications, the moreaccurate can the estimation of the 3D shape, volume or surface(s) of thebone be.

The 3D shape, volume or surface of the bone can, for example, beestimated by filling in the information, e.g. intermediate or connectinglandmarks, distances, dimensions, surfaces, edges, angles, axes,curvatures, shapes, lengths, widths, depths and/or other featuresbetween known landmarks, distances, dimensions, surfaces, edges, angles,axes, curvatures, shapes, lengths, widths, depths and/or other featuresderived from the one, two, three or more x-ray images. In someembodiments of the invention, a standard model of the bone can be usedand can be deformed using one or more of the known landmarks, distances,dimensions, surfaces, edges, angles, axes, curvatures, shapes, lengths,widths, depths and/or other features derived from the one, two, three ormore x-ray images. Such deformations can be performed using variousstatistical models known in the art.

In some embodiments of the invention, a database or library of bonemodels and tissue models can be used. The one or more of these anatomiclandmarks, distances, dimensions, surfaces, edges, angles, axes,curvatures, shapes, lengths, widths, depths and/or other features can beused to identify a standard bone shape and/or a standard cartilage shapeby comparing the one or more landmarks, distances, dimensions, surfaces,edges, angles, axes, curvatures, shapes, lengths, widths, depths and/orother features with data in a reference database of reference patientsand/or reference bone and/or cartilage shapes and by selecting a 3Dmodel that most closely matches the selected landmarks, distances,dimensions, surfaces, edges, angles, axes, curvatures, shapes, lengths,widths, depths and/or other features. In this manner, the 3D shape ofthe patient's bones and/or cartilage, e.g. the distal femur and/or theproximal tibia and/or the acetabulum and/or the proximal femur, and/orthe vertebral body and/or the spinal elements and/or the glenoid and/orthe proximal humerus, can be estimated without the need acquire 3D dataor without the need of segmentation of the 3D data or limiting theamount of segmentation needed from available 3D data, e.g. a CT scan oran MRI scan of the patient. The reference database can be, for example,an anatomic reference database from cadaver data. The reference databasecan also be, for example, scan data, e.g. acquired in the NIHOsteoarthritis Initiative or acquired from imaging data to generatepatient specific instruments for knee replacement. Such scan data can beused to generate a database of 3D shapes of patients with different age,gender, ethnic background, race, weight, height and/or BMI.

Of note, the use 2D imaging data or 3D imaging data, e.g. x-ray,ultrasound, CT or MRI, in combination with one or more referencedatabases of 3D shape(s) of select anatomic structures, such as a bone,a cartilage, an organ for reducing or limiting or obviating the need foracquiring 3D data or for segmenting 2D or 3D data is applicable to anyembodiment of the invention throughout the specification including forall other clinical applications, e.g. hip replacement, knee replacement,spinal surgery, spinal fusion, vertebroplasty, kyphoplasty, fracturefixation, brain surgery, liver surgery, cancer surgery etc.

In some embodiments of the invention, a standard model, optionallyalready deformed using the patient's landmarks, distances, dimensions,surfaces, edges, angles, axes, curvatures, shapes, lengths, widths,depths and/or other features, can be combined or fused with a modelselected from a database using the patient's landmarks, distances,dimensions, surfaces, edges, angles, axes, curvatures, shapes, lengths,widths, depths and/or other features. In some embodiments, the modelselected from the database can be deformed and/or adapted using thepatient's landmarks, distances, dimensions, surfaces, edges, angles,axes, curvatures, shapes, lengths, widths, depths and/or other features.Such deformations can be performed using various statistical modelsknown in the art.

If one or more x-rays are used, they can, for example, be obtained in anAP projection of the knee (or PA), and a lateral projection of the knee.Other views are possible, as known in the art, e.g. a tunnel view,Merchant view, patellar view, oblique views, standing views, supineviews, prone views. Optionally, the medial and lateral femoral condylescan be identified on the AP/PA and/or lateral and/or oblique views;optionally, the medial and lateral tibial plateau can be identified onthe AP/PA and/or lateral and/or oblique views. Other landmarks,distances, dimensions, surfaces, edges, angles, axes, curvatures,shapes, lengths, widths, depths and/or other features can be identified.

A lateral knee x-ray can, for example, be used to derive curvatureinformation about the medial and the lateral condyle. Two distinctcurves can be seen on a lateral knee radiograph, one representing themedial condyle and the other representing the lateral condyle. In mostinstances, the lateral condyle has a smaller radius than the medialcondyle, for example in the central weight-bearing zone. Software canidentify and/or segment each curve using, for example, some of thesoftware packages described in Data Segmentation. This can be followedby a curvature analysis assessing the radii of each curve. In someembodiments of the invention, the curve with the smaller radii, e.g. inthe central weight bearing area, can be assigned as the lateral condyle.Other combinations are possible.

The foregoing description of techniques to estimate or morph thethree-dimensional shape of a patient's bone is only exemplary in natureand is in no way meant to be limiting of the invention. Someone skilledin the art will readily recognize other means to estimate the shape ofthe patient's bone in three dimensions. Any technique known in the artfor determining or estimating the three-dimensional shape of a bone fromtwo-dimensional data can be used. Any technique known in the art formodeling and displaying the three-dimensional shape of a bone fromtwo-dimensional data can be used.

Bone and/or Tissue Morphing Using Mechanical Probes and/orOpto-Electronic and/or RF Probes

In some embodiments of the invention, a mechanical probe can be used todetermine the three-dimensional shape of a patient's tissue, e.g.cartilage or bone or organ tissue, intra-operatively. The tissue probecan be attached to a stand or holder. The tissue probe can also behandheld.

The tissue probe can be configured similar to a mechanical detectiondevice known in the art and used, for example, for industrial shapeinspection purposes, e.g. coordinate measuring machines (CMM) known inthe art, such as, for example, the Faro arm system.

In some embodiments of the invention, a mechanical probe can be usedthat has at least one of an optical marker, navigation marker, includinginfrared markers, retroreflective markers, RF markers, LED and/or IMUattached. The position and/or orientation and/or alignment and/ordirection of movement of the probe can be determined then, for example,using a navigation system and/or an image and/or video capture systemintegrated into, attached to or separate from the OHMD.

By moving the mechanical probe along the bone, cartilage, tissue and/ororgan surface, the position of the tip of the probe can, for example, beregistered and, for example, a point cloud can be generated which can beused to generate a 3D surface. Standard techniques known in the art,e.g. tessellation, can be used for this purpose.

Bone and/or Tissue Morphing Using Optical Probes and/or 3D Scannersand/or Image Capture Systems

In some embodiments of the invention, an image and/or video capturesystem integrated into, attached to or separate from the OHMD can beused to image the patient's bone and/or cartilage and/or tissue and/orligaments and/or menisci and/or organ surface. With the position,orientation, alignment and/or direction of movement of the image and/orvideo capture system(s) known, for example using optical markers,navigation markers including infrared markers, retroreflective markers,RF markers, LEDs and/or IMUs, images of the patient's bone and/orcartilage and/or tissue and/or ligaments and/or menisci and/or organsurface can be acquired from multiple viewpoints or continuously and,using software and image processing as described in Data Segmentation orspatial mapping techniques as described in Spatial Mapping, images canbe used to derive one or more 3D volumes, 3D surfaces and/or 3D shapesof the patient's bone and/or cartilage and/or tissue and/or ligamentsand/or menisci and/or organ. The accuracy of such image acquisitions andreconstruction of 3D volumes, 3D surfaces and/or 3D shapes canoptionally be enhanced with image and/or video capture systems that usetwo or more cameras, which can be used to generated parallax informationand/or stereoscopic information of the same structures, wherein, forexample, the parallax and/or stereoscopic information can be used toenhance the accuracy of the reconstructions. Alternatively, theinformation from two or more cameras can be merged by averaging the 3Dcoordinates or detected surface points or other geometric structuressuch as planes or curved surfaces.

In some embodiments of the invention, 3D laser scanners or depth sensorsknown in the art, such as, for example, the Structure laser scannerprovided by Occipital Inc., can be used to image the surface of thepatient's bone and/or cartilage and/or tissue and/or ligaments and/ormenisci and/or organ. Other 3D scanners known in the art can be used.Any laser scanning, optical or light scanning technique known in the artfor determining, estimating or deriving the 3D volume, 3D surface or 3Dshape of a structure known in the art can be used.

In some embodiments of the invention, the 3D scanner or image and/orvideo capture system can be attached to an arm or tripod. Images of thepatient's bone and/or cartilage and/or tissue and/or ligaments and/ormenisci and/or organ can be acquired at a constant distance. Images ofthe patient's bone and/or cartilage and/or tissue and/or ligamentsand/or menisci and/or organ can be acquired at a variable distance. Thelaser or optical scanner can optionally be used to measure the distanceto the patient's bone and/or cartilage and/or tissue and/or ligamentsand/or menisci and/or organ during the image acquisition. Using thelaser's starting position or the starting position of the image and/orvideo capture system and/or at least one of an optical marker,navigation marker including infrared markers, retroreflective markers,RF markers, LED and/or IMU, the position, orientation, alignment and/ordirection of movement of the image and/or video capture system and/or 3Dscanner can be known throughout the acquisition allowing formagnification correction and optional view angle adjustments and/orprojection and/or surface generation calculation and/or adjustmentsand/or corrections.

Combining Pre-Operative and Intra-Operative Data

In some embodiments of the invention, 2D or 3D data obtainedintra-operatively with a mechanical probe, opto-electronic probe, RFprobe, optical probe, image and/or video capture system, laser scannerand/or 3D scanner can be combined with pre-operative data, e.g.pre-operative imaging data and/or a virtual surgical plan.

The 2D or 3D information obtained pre-operatively can, for example,include mechanical axis information, e.g. of the knee and/or lowerextremity (e.g. obtained using a standing x-ray), rotation axisinformation, e.g. of a hip or a knee, e.g. using epicondylar axisinformation, posterior condylar axis information, tibial tubercleinformation, one or more AP dimensions of a joint, one or more MLdimensions of a joint, one or more SI dimensions of a joint, a medialcondyle curvature and/or a lateral condyle curvature, e.g. as seen on alateral and/or an AP radiograph, a medial tibial curvature and/or alateral tibial curvature, e.g. as seen on a lateral and/or an APradiograph, joint line information, e.g. the location of a medial and/ora lateral joint line in a knee, offset information, e.g. an offset in ahip or an offset between a medial and/or a lateral condyle.

The 2D or 3D data obtained intra-operatively can, for example, includedimensional information, geometric information, curvature information,volume information, shape information, and/or surface information of thetissue, organ, e.g. cartilage and/or bone. The 2D or 3D data obtainedintra-operatively can, for example, include information about joint linelocation, e.g. medial and/or lateral, femoral offsets and/or tibialoffsets, measured based on cartilage and/or subchondral bone.

Optionally, adjustments or corrections can be applied to data obtainedpre-operatively and/or intra-operatively. For example, osteophytesand/or subchondral cysts can be virtually removed from the pre-operativeand/or intra-operative 2D or 3D data. Flattening of a joint surface seenon any of the data can be optionally corrected, e.g. by applying acorrected shape, e.g. using spline surfaces or smoothing functions oraveraging functions.

In some embodiments of the invention, 2D or 3D pre-operative data can becombined with 2D or 3D intra-operative data. For example, mechanicalaxis information obtained from a pre-operative standing x-ray can becombined with an intra-operative 3D scan of a joint, e.g. a knee jointor a hip joint. A virtual surgical plan can be developed or derivedbased on the combined data, for example with resections that are plannedto maintain or restore normal mechanical axis alignment or any otheralignment desired by the surgeon, e.g. 5% or less of varus or valgusalignment of a joint. If a virtual surgical plan has already beendeveloped pre-operatively, the virtual surgical plan can be modifiedintra-operatively using intra-operative 3D scan information of one ormore joints, for example using more accurate intra-operative surfaceinformation of the joint or organ.

In some embodiments of the invention, 3D surfaces morphed from 2Dpre-operative data, e.g. using one or more pre-operative x-rays, can becombined with 3D surfaces derived intra-operatively, e.g. derived usingan intra-operative mechanical and/or opto-electronic and/or laser and/or3D scanner. For example, the pre-operative morphed surfaces of a femoralhead can be matched, aligned, superimposed or merged in this manner withthe intra-operative surfaces. Or the pre-operative morphed surfaces ofone or both femoral condyles and/or tibial plateaus can be matched,aligned superimposed or merged in this manner with their correspondingintra-operative surfaces. By matching, aligning, superimposing ormerging surfaces derived from pre-operative and intra-operative data,axis information obtained on pre-operative data, e.g. standing x-rayscan be readily superimposed or merged with intra-operative data. Theresultant model can be used to develop, derive and/or modify a virtualsurgical plan, for example with subsequent display of one or more cutplanes or tissue resections or axes by an OHMD.

2D data obtained pre-operatively and/or intra-operatively using 2D to 3Dtissue morphing, e.g. bone morphing, for example as described in thespecification, and morphed into a 3D model can be displayedstereoscopically and/or non-stereoscopically using one or more OHMDdisplays. In addition, any of a virtual surgical tool, virtual surgicalinstrument including a virtual surgical guide or cut block, virtualtrial implant, virtual implant component, virtual implant or virtualdevice, a predetermined start point, predetermined start position,predetermined start orientation or alignment, predetermined intermediatepoint(s), predetermined intermediate position(s), predeterminedintermediate orientation or alignment, predetermined end point,predetermined end position, predetermined end orientation or alignment,predetermined path, predetermined plane, predetermined cut plane,predetermined contour or outline or cross-section or surface features orshape or projection, predetermined depth marker or depth gauge,predetermined angle or orientation or rotation marker, predeterminedaxis, e.g. rotation axis, flexion axis, extension axis, predeterminedaxis of the virtual surgical tool, virtual surgical instrument includingvirtual surgical guide or cut block, virtual trial implant, virtualimplant component, implant or device, non-visualized portions for one ormore devices or implants or implant components or surgical instrumentsor surgical tools, and/or one or more of a predetermined tissue changeor alteration can be displayed by the OHMD concurrent with the 2D to 3Dmorphed 3D model, e.g. bone model, stereoscopically ornon-stereoscopically. The one or more of a virtual surgical tool,virtual surgical instrument including a virtual surgical guide or cutblock, virtual trial implant, virtual implant component, virtual implantor virtual device, predetermined start point, predetermined startposition, predetermined start orientation or alignment, predeterminedintermediate point(s), predetermined intermediate position(s),predetermined intermediate orientation or alignment, predetermined endpoint, predetermined end position, predetermined end orientation oralignment, predetermined path, predetermined plane, predetermined cutplane, predetermined contour or outline or cross-section or surfacefeatures or shape or projection, predetermined depth marker or depthgauge, predetermined angle or orientation or rotation marker,predetermined axis, e.g. rotation axis, flexion axis, extension axis,predetermined axis of the virtual surgical tool, virtual surgicalinstrument including virtual surgical guide or cut block, virtual trialimplant, virtual implant component, implant or device, non-visualizedportions for one or more devices or implants or implant components orsurgical instruments or surgical tools, and/or one or more of apredetermined tissue change or alteration can be planned using the 2D to3D morphed 3D model, for example using a virtual surgical plan.

In some embodiments of the invention, at least one or more of the samelandmarks, distances, dimensions, surfaces, edges, angles, axes,curvatures, shapes, lengths, widths, depths and/or other features usedfor 2D to 3D tissue morphing, e.g. bone morphing, can be used forintra-operative registration of live data and virtual data, e.g.pre-operative data, of the patient by identifying the at least one ormore of the corresponding landmarks, distances, dimensions, surfaces,edges, angles, axes, curvatures, shapes, lengths, widths, depths and/orother features in the live data, using, for example, some of thetechniques described in the specification.

FIG. 15 is an example how 2D to 3D morphed data can be used or applied.The example is in no way meant to be limiting of the invention. In thisexample, 2D x-ray images can be obtained, optionally with multipleprojections 165. One or more tissues, e.g. bone, cartilage, theirlandmarks, shapes and or geometries or other features can be derived 166and can be optionally adjusted 167. Interconnecting or fill features canbe determined 168, a closely fitting or matching model can be selectedfrom a library or database of models 169, a standard model can beselected and optionally be deformed 170 using the shapes, geometries orfeatures 166, a closely fitting or matching model can be selected from alibrary or database of models 171 and deformed using the information in166. Steps and processes in 168, 169, 170, and 171 can optionally becombined 172. Steps and processes 168, 169, 170, 171, and 172 can beused to generate a 2D to 3D morphed model 173, which can be used togenerate pre-operative virtual surgical plan 174. The morphed model 173and the pre-operative virtual surgical plan 174 can be displayed by oneor more OHMDs 175, optionally stereoscopic 176 or non-stereoscopic 177.An intra-operative virtual surgical plan 179 can optionally besuperimposed, merged, matched or aligned with the pre-operative virtualsurgical plan 174. An intra-operative scan or probe data 180 can be usedto generate a model of the patient using intra-operative data, e.g. 2D,2D to 3D morphed, 3D 181, which can optionally be superimposed, matched,merged or aligned 173 with the morphed model of the patient usingpre-operative data 173 or the pre-operative virtual surgical plan 174.Optional adjustments to the model of the patient using intra-operativedata 181 can be made 182.

Virtual Data and Live Data Seen Through One or More OHMDs

A virtual surgical plan using, for example, virtual data of the patient,can be used to develop or determine any of the following for placing ordirecting a surgical tool, a surgical instrument, a trial implantcomponent, a trial implant, an implant component, an implant, a deviceincluding any type of biological treatment or implant or matrix known inthe art:

-   -   Predetermined start point    -   Predetermined start position    -   Predetermined start orientation/alignment    -   Predetermined intermediate point(s)    -   Predetermined intermediate position(s)    -   Predetermined intermediate orientation/alignment    -   Predetermined end point    -   Predetermined end position    -   Predetermined intermediate orientation/alignment    -   Predetermined path    -   Predetermined plane (e.g. for placing or orienting a surgical        instrument or an implant component)    -   Predetermined cut plane (e.g. for directing a saw or other        surgical instruments (e.g. drills, pins, cutters, reamers,        rasps, impactors, osteotomes) and/or for placing or orienting an        implant component or a trial implant component)    -   Projected contour/outline/cross-section/surface        features/shape/projection    -   Predetermined depth marker or depth gauge, optionally        corresponding to a physical depth marker or depth gauge on the        physical surgical tool, surgical instrument, trial implant,        implant component, implant or device    -   Predetermined angle/orientation/rotation marker, optionally        corresponding to a physical angle/orientation/rotation marker on        the physical surgical tool, surgical instrument, trial implant,        implant component, implant or device    -   Predetermined axis, e.g. rotation axis, flexion axis, extension        axis    -   Predetermined axis of the physical surgical tool, surgical        instrument, trial implant, implant component, implant or device,        e.g. a long axis, a horizontal axis, an orthogonal axis, a        drilling axis, a pinning axis, a cutting axis (move this to        ensuing examples)    -   Estimated/projected non-visualized portions of        device/implant/implant component/surgical instrument/surgical        tool, e.g. using image capture or markers attached to        device/implant/implant component/surgical instrument/surgical        tool with known geometry    -   Predetermined virtual tissue change/alteration.

Any of the foregoing, e.g. a cut plane or an outline, e.g. an outline ofan implant or a surgical instrument, can be displayed in 2D and/or in3D, optionally alternatingly. For example, a 2D visualization, e.g. aline, of a cut plane can be used when a surgeon looks substantially onend on a bone, e.g. a distal femur, for orienting and/or directing acutting instrument, e.g. a saw or a saw blade. When the surgeon looksfrom the side, e.g. at an angle, the visualization can optionally switchto a 3D display to show the desired angular orientation of the cutand/or the blade in relationship to the bone. The display can alsoremain in 2D mode. The switching between 2D and 3D display can bemanual, e.g. through a voice command or a command on a virtuallyprojected keyboard or a virtually projected user interface, orautomatic, e.g. based on the position and/or orientation of theoperator's head and/or the OHMD in relationship to the surgical site(e.g. operator head/OHMD in frontal orientation relative to surgicalsite, or close to including 90 degree side (near orthogonal)orientation, or angular, non-90 degree side orientation, e.g. 30, 40,50, 60, 70 degree angles). A 2D or 3D display of a cut plane can helpdetermine/display the desired angular orientation of the intended cut.The angular orientation can, for example, be a reflection of aplanned/intended mechanical axis correction in a knee replacement, aplanned/intended femoral component flexion or extension in a kneereplacement, a planned/intended tibial slope in a knee replacement or aplanned/intended femoral neck resection for a planned/intended leglength in a hip replacement.

A 2D or 3D display can also include multiple cut planes, e.g. two ormore femoral neck cuts in a hip replacement procedure, as can be used inhip replacement procedures involving, for example, an anterior approachand using a “napkin ring” like dual cut through the femoral neck. Inthis example, the 3D cut plane can include the distal cut plane at itsinferior pointing surface and the proximal cut plane at its superiorsurface. These “napkin ring” inferior, distal facing, and superior,proximal facing cuts can be parallel or non-parallel, e.g. for easierextraction of the femoral head. Any cut planes visualized in 2D or 3Dusing the OHMD display can be parallel or non-parallel, usingstereoscopic or non-stereoscopic display.

If the surgeon elects to change or adjust any of a virtual surgicaltool, virtual surgical instrument including a virtual surgical guide orcut block, virtual trial implant, virtual implant component, virtualimplant or virtual device, a predetermined start point, predeterminedstart position, predetermined start orientation or alignment,predetermined intermediate point(s), predetermined intermediateposition(s), predetermined intermediate orientation or alignment,predetermined end point, predetermined end position, predetermined endorientation or alignment, predetermined path, predetermined plane,predetermined cut plane, predetermined contour or outline orcross-section or surface features or shape or projection, predetermineddepth marker or depth gauge, predetermined angle or orientation orrotation marker, predetermined axis, e.g. rotation axis, flexion axis,extension axis, predetermined axis of the virtual surgical tool, virtualsurgical instrument including virtual surgical guide or cut block,virtual trial implant, virtual implant component, implant or device,non-visualized portions for one or more devices or implants or implantcomponents or surgical instruments or surgical tools, and/or one or moreof a predetermined tissue change or alteration used in the one or morevirtual surgical plans using, for example, a virtual interface displayedby the OHMD display, e.g. a finger slider or finger tab to move and/orrotate a virtual cut plane by virtually touching it, or any otherinterface, including, for example, a finger command or a voice command,the virtual representation of the virtual data can move accordingly andthe virtual data displayed in the OHMD can be updated accordingly in thesurgeon's display. The change in position and/or orientation of thevirtual representation of the virtual data can also be seen in otherOHMDs, e.g. worn by a second surgeon, a resident, a scrub nurse or a PA,and the projection of the virtual data can also be updated accordinglyin a second, third or any additional OHMD units used, for example, by asecond surgeon, a resident, a scrub nurse or a PA during the surgery.Optionally, the virtual interface or any other interface to change oradjust one or more of the virtual data can only be available for thesurgeon's OHMD unit, i.e. the lead OHMD unit, while the other OHMD unitscan operate as slave units that simply follow the display of the leadOHMD unit. In this manner, potential intraoperative errors, for examplewith a non-surgeon modifying virtual data or aspects of the virtualsurgical plan, can be avoided. Optionally, the lead can be passed overto any of the other units, in which case the surgeon's OHMD unit canoperate as a slave unit. This can be beneficial when complex changes arerequired to the virtual surgical plan and/or the virtual data of thepatient, which may require a separate person to implement such changes,while the surgeon is managing the physical operation in the livepatient.

In some embodiments of the invention, the OHMD unit of the surgeon cancapture the live data of the patient using one or more image and/orvideo capture systems integrated into or attached to the OHMD. Thecaptured live data of the patient can then be transmitted in electronic,digital form as live stream to slave OHMD units, optionally togetherwith the virtual data of the patient, e.g. superimposed onto orco-displayed with the virtual data of the patient. Alternatively, theslave units in this example can be non-see through virtual reality (VR)systems such as the Google Daydream system or the Zeiss VR One systemand others known in the art. Any intended cut plane displayed by theOHMD can optionally include or account for the thickness of the sawblade to reflect bone last during the sawing step. Any intended path fora drill or pin or other surgical instrument can include or account forthe thickness of the surgical instrument to reflect bone lost during thesurgical step. In addition, any bone lost due to movement of a surgicalinstrument, e.g. movement not in the primary direction of the surgicalstep such as saw blade flutter or saw vibration or a slightly eccentricdrill or drill vibration can be included in the virtual surgical plan,for example through estimations of saw blade flutter or saw vibrationsin addition to a known saw blade thickness, and can be accounted for inthe virtual resection planning and in the resultant display of one ormore 2D or 3D cut planes by the OHMD.

Someone skilled in the art can readily recognize that accounting for thethickness of a saw blade or dimensions of other bone removinginstruments as well as related instrument or device movement orvibration induced bone loss can be accounted for in one, two, three ormore bone removing steps, if a surgical procedure involves multiple boneremoving steps, such as the femoral preparation of a partial or totalknee replacement, which can include two, three or more bone cuts.

When the OHMD is used to display the estimated/projected non-visualizedportions of a device, an implant, an implant component, a surgicalinstrument and/or a surgical tool, the display of the non-visualizedportion of the device, implant, implant component, surgical instrumentand/or surgical tool can also account for any bone loss that may havebeen or will be induced by the device, implant, implant component,surgical instrument and/or surgical tool. By accounting for the boneloss induced by the device, implant, implant component, surgicalinstrument and/or surgical tool, the virtual surgical plan and thedisplay of any surgical steps including subsequent surgical steps by theOHMD can be more accurate.

A virtual surgical plan can be used to define a predetermined startpoint for a surgical tool, a surgical instrument, a trial implantcomponent, a trial implant, an implant component, an implant, a device.A start point can be, for example, the entry at the patient's skin. Ifpre-operative imaging, e.g. ultrasound, CT and/or MRI, is used fordeveloping the surgical plan, the skin can be located in the imagingdata and the start point can be defined at an area typically near theintended surgical site. A start point can also be defined at a selectsoft-tissue depth, e.g. 5, 8 or 10 cm into the soft-tissue, e.g.subcutaneous tissue or muscle or other tissues or organ tissue. A startpoint can be defined at the surface of an organ, e.g. a liver or aspleen or a kidney or a bladder or a brain. A start point can be definedat an anatomic landmark or in relationship to an anatomic landmark of anorgan, e.g. a rim of a liver, a liver portal, an entry of an inferiorvena cava into the liver, an entry of a portal vein into the liver, asuperior or inferior pole of a kidney, a renal hilum. A start point canbe defined at a bone surface or bony landmark The one or more of avirtual surgical tool, virtual surgical instrument including a virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, virtual implant or virtual device, one or more apredetermined start point, predetermined start position, predeterminedstart orientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration used in the one or more virtual surgical plans can behighlighted in the one or more OHMD displays using various techniquesknown in the art, including but not limited to:

-   -   Colored display    -   Grey scale display    -   Shaded display    -   Patterned display, e.g. squares, lines, bars    -   Line display, e.g. solid, stippled, dotted    -   Arrow display    -   Target like display    -   Intermittent display, e.g. blinking or flashing    -   Appearing or disappearing display    -   Magnified display    -   Minified display

For example, a virtual surgical tool, virtual surgical instrumentincluding a virtual surgical guide or cut block, virtual trial implant,virtual implant component, virtual implant or virtual device, apredetermined start point, predetermined start position, predeterminedstart orientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration is displayed by the OHMD multiple colors can be chosen.

For example, a virtual surgical tool, virtual surgical instrumentincluding a virtual surgical guide or cut block, virtual trial implant,virtual implant component, virtual implant or virtual device, apredetermined start point, predetermined start position, predeterminedstart orientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration can be highlighted using an arrow display. The arrows can bealigned with the direction of the one or more surgical tools, surgicalinstruments, implant components, implants or devices. The arrows canalso not be aligned with the direction of the one or more surgicaltools, surgical instruments, implant components, implants or devices.The arrows can be orthogonal to the direction of the one or moresurgical tools, surgical instruments, implant components, implants ordevices. The arrows can be aligned with the one or more surgical tools,surgical instruments, implant components, implants or devices. Thearrows cannot be orthogonal with the one or more surgical tools,surgical instruments, implant components, implants or devices.

One or more arrows can directly point at the one or more of a virtualsurgical tool, virtual surgical instrument including a virtual surgicalguide or cut block, virtual trial implant, virtual implant component,virtual implant or virtual device, one or more a predetermined startpoint, predetermined start position, predetermined start orientation oralignment, predetermined intermediate point(s), predeterminedintermediate position(s), predetermined intermediate orientation oralignment, predetermined end point, predetermined end position,predetermined end orientation or alignment, predetermined path,predetermined plane, predetermined cut plane, predetermined contour oroutline or cross-section or surface features or shape or projection,predetermined depth marker or depth gauge, predetermined angle ororientation or rotation marker, predetermined axis, e.g. rotation axis,flexion axis, extension axis, predetermined axis of the virtual surgicaltool, virtual surgical instrument including virtual surgical guide orcut block, virtual trial implant, virtual implant component, implant ordevice, non-visualized portions for one or more devices or implants orimplant components or surgical instruments or surgical tools, and/or oneor more of a predetermined tissue change or alteration. The one or morearrows can optionally be magnified or minified. The one or more arrowscan optionally be displayed intermittently, e.g. blinking or flashing.The one or more arrows can optionally be appearing or disappearing. Forexample, the one or more arrows can disappear when the predetermined endpoint is reached by the physical surgical tool, surgical instrument,implant component, implant or device.

The one or more of a virtual surgical tool, virtual surgical instrumentincluding a virtual surgical guide or cut block, virtual trial implant,virtual implant component, virtual implant or virtual device, one ormore predetermined start point, predetermined start position,predetermined start orientation or alignment, predetermined intermediatepoint(s), predetermined intermediate position(s), predeterminedintermediate orientation or alignment, predetermined end point,predetermined end position, predetermined end orientation or alignment,predetermined path, predetermined plane, predetermined cut plane,predetermined contour or outline or cross-section or surface features orshape or projection, predetermined depth marker or depth gauge,predetermined angle or orientation or rotation marker, predeterminedaxis, e.g. rotation axis, flexion axis, extension axis, predeterminedaxis of the virtual surgical tool, virtual surgical instrument includingvirtual surgical guide or cut block, virtual trial implant, virtualimplant component, implant or device, non-visualized portions for one ormore devices or implants or implant components or surgical instrumentsor surgical tools, and/or one or more of a predetermined tissue changeor alteration can be highlighted using a target like display. More thanone target-like display can be used.

The target-like display can, for example, be positioned over a startingpoint, one or more intermediate points, an end point, a startingposition, one or more intermediate positions, an end position, aintended path, predetermined plane, predetermined cut plane, apredetermined axis of the physical surgical tool, surgical instrument,trial implant, implant component, implant or device. A line or an axisoriented in orthogonal fashion through the target and passing throughthe center of one or more targets can optionally be aligned with apredetermined path, predetermined plane, predetermined cut plane, orpredetermined axis of the physical surgical tool, surgical instrument,trial implant, implant component, implant or device, and/or one or moreof a predetermined tissue change/alteration.

An intermittent, e.g. blinking or flashing display can be used to showone or more of a virtual surgical tool, virtual surgical instrumentincluding a virtual surgical guide or cut block, virtual trial implant,virtual implant component, virtual implant or virtual device, one ormore of a predetermined start point, predetermined start position,predetermined start orientation or alignment, predetermined intermediatepoint(s), predetermined intermediate position(s), predeterminedintermediate orientation or alignment, predetermined end point,predetermined end position, predetermined end orientation or alignment,predetermined path, predetermined plane, predetermined cut plane,predetermined contour or outline or cross-section or surface features orshape or projection, predetermined depth marker or depth gauge,predetermined angle or orientation or rotation marker, predeterminedaxis, e.g. rotation axis, flexion axis, extension axis, predeterminedaxis of the virtual surgical tool, virtual surgical instrument includingvirtual surgical guide or cut block, virtual trial implant, virtualimplant component, implant or device, non-visualized portions for one ormore devices or implants or implant components or surgical instrumentsor surgical tools, and/or one or more of a predetermined tissue changeor alteration. An intermittent display can, for example, highlight ifand when one or more of the surgical tool, surgical instrument, trialimplant, implant component, implant or device are not aligned with oneor more of the virtual surgical tool, virtual surgical instrumentincluding a virtual surgical guide or cut block, virtual trial implant,virtual implant component, virtual implant or virtual device, one ormore of the predetermined start point, predetermined start position,predetermined start orientation or alignment, predetermined intermediatepoint(s), predetermined intermediate position(s), predeterminedintermediate orientation or alignment, predetermined end point,predetermined end position, predetermined end orientation or alignment,predetermined path, predetermined plane, predetermined cut plane,predetermined contour or outline or cross-section or surface features orshape or projection, predetermined depth marker or depth gauge,predetermined angle or orientation or rotation marker, predeterminedaxis, e.g. rotation axis, flexion axis, extension axis, predeterminedaxis of the virtual surgical tool, virtual surgical instrument includingvirtual surgical guide or cut block, virtual trial implant, virtualimplant component, implant or device, non-visualized portions for one ormore devices or implants or implant components or surgical instrumentsor surgical tools, and/or one or more of a predetermined tissue changeor alteration. An intermittent display can, for example, highlight ifand when one or more of the surgical tool, surgical instrument, trialimplant, implant component, implant or device are aligned with one ormore of the one or more of a virtual surgical tool, virtual surgicalinstrument including a virtual surgical guide or cut block, virtualtrial implant, virtual implant component, virtual implant or virtualdevice, a predetermined start point, predetermined start position,predetermined start orientation or alignment, predetermined intermediatepoint(s), predetermined intermediate position(s), predeterminedintermediate orientation or alignment, predetermined end point,predetermined end position, predetermined end orientation or alignment,predetermined path, predetermined plane, predetermined cut plane,predetermined contour or outline or cross-section or surface features orshape or projection, predetermined depth marker or depth gauge,predetermined angle or orientation or rotation marker, predeterminedaxis, e.g. rotation axis, flexion axis, extension axis, predeterminedaxis of the virtual surgical tool, virtual surgical instrument includingvirtual surgical guide or cut block, virtual trial implant, virtualimplant component, implant or device, non-visualized portions for one ormore devices or implants or implant components or surgical instrumentsor surgical tools, and/or one or more of a predetermined tissue changeor alteration.

An intermittent display can optionally change colors or haveintermittent, varying color schemes. For example, a blinking or flashingred color can turn into solid, not intermittent green color when one ormore of the physical surgical tool, surgical instrument, trial implant,implant component, implant and/or devices are aligned with one or moreof a virtual surgical tool, virtual surgical instrument including avirtual surgical guide or cut block, virtual trial implant, virtualimplant component, virtual implant or virtual device, or one or more ofthe predetermined start point, predetermined start position,predetermined start orientation or alignment, predetermined intermediatepoint(s), predetermined intermediate position(s), predeterminedintermediate orientation or alignment, predetermined end point,predetermined end position, predetermined end orientation or alignment,predetermined path, predetermined plane, predetermined cut plane,predetermined contour or outline or cross-section or surface features orshape or projection, predetermined depth marker or depth gauge,predetermined angle or orientation or rotation marker, predeterminedaxis, e.g. rotation axis, flexion axis, extension axis, predeterminedaxis of the virtual surgical tool, virtual surgical instrument includingvirtual surgical guide or cut block, virtual trial implant, virtualimplant component, implant or device, non-visualized portions for one ormore devices or implants or implant components or surgical instrumentsor surgical tools, and/or one or more of a predetermined tissue changeor alteration.

An intermittent display can, for example, highlight if and when one ormore of the surgical tool, surgical instrument, trial implant, implantcomponent, implant or device are not aligned with one or more of thepredetermined start point, predetermined start position, predeterminedstart orientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration in the OHMD can turn from a solid color, e.g. green or blue,to a blinking or flashing red color. Different colors can be chosen forintermediate versus final, end positions, e.g. blue for intermediate andgreen for final/end.

An appearing or disappearing display can be used to show one or more ofa predetermined start point, predetermined start position, predeterminedstart orientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration and/or one or more of a predetermined position and/ororientation of the virtual surgical tool, virtual surgical instrumentincluding virtual surgical guide or cut block, virtual trial implant,virtual implant component, implant or device inside the OHMD. Anappearing or disappearing display can, for example, highlight if andwhen one or more of the surgical tool, surgical instrument, trialimplant, implant component, implant or device are not aligned with oneor more of a predetermined start point, predetermined start position,predetermined start orientation or alignment, predetermined intermediatepoint(s), predetermined intermediate position(s), predeterminedintermediate orientation or alignment, predetermined end point,predetermined end position, predetermined end orientation or alignment,predetermined path, predetermined plane, predetermined cut plane,predetermined contour or outline or cross-section or surface features orshape or projection, predetermined depth marker or depth gauge,predetermined angle or orientation or rotation marker, predeterminedaxis, e.g. rotation axis, flexion axis, extension axis, predeterminedaxis of the virtual surgical tool, virtual surgical instrument includingvirtual surgical guide or cut block, virtual trial implant, virtualimplant component, implant or device, non-visualized portions for one ormore devices or implants or implant components or surgical instrumentsor surgical tools, and/or one or more of a predetermined tissue changeor alteration and/or one or more of a predetermined position and/ororientation of the virtual surgical tool, virtual surgical instrumentincluding virtual surgical guide or cut block, virtual trial implant,virtual implant component, implant or device. In this example, the oneor more of a predetermined start point, predetermined start position,predetermined start orientation or alignment, predetermined intermediatepoint(s), predetermined intermediate position(s), predeterminedintermediate orientation or alignment, predetermined end point,predetermined end position, predetermined end orientation or alignment,predetermined path, predetermined plane, predetermined cut plane,predetermined contour or outline or cross-section or surface features orshape or projection, predetermined depth marker or depth gauge,predetermined angle or orientation or rotation marker, predeterminedaxis, e.g. rotation axis, flexion axis, extension axis, predeterminedaxis of the virtual surgical tool, virtual surgical instrument includingvirtual surgical guide or cut block, virtual trial implant, virtualimplant component, implant or device, non-visualized portions for one ormore devices or implants or implant components or surgical instrumentsor surgical tools, and/or one or more of a predetermined tissue changeor alteration and/or one or more of a predetermined position and/ororientation of the virtual surgical tool, virtual surgical instrumentincluding virtual surgical guide or cut block, virtual trial implant,virtual implant component, implant or device can appear in the OHMDdisplay when the physical surgical tool, surgical instrument, trialimplant, implant component, implant, and/or device are not aligned, e.g.with the surgical plan or the one or more of the predetermined startpoint, predetermined start position, predetermined start orientation oralignment, predetermined intermediate point(s), predeterminedintermediate position(s), predetermined intermediate orientation oralignment, predetermined end point, predetermined end position,predetermined end orientation or alignment, predetermined path,predetermined plane, predetermined cut plane, predetermined contour oroutline or cross-section or surface features or shape or projection,predetermined depth marker or depth gauge, predetermined angle ororientation or rotation marker, predetermined axis, e.g. rotation axis,flexion axis, extension axis, predetermined axis of the virtual surgicaltool, virtual surgical instrument including virtual surgical guide orcut block, virtual trial implant, virtual implant component, implant ordevice, non-visualized portions for one or more devices or implants orimplant components or surgical instruments or surgical tools, and/or oneor more of a predetermined tissue change or alteration and/or one ormore of a predetermined position and/or orientation of the virtualsurgical tool, virtual surgical instrument including virtual surgicalguide or cut block, virtual trial implant, virtual implant component,implant or device. The one or more of a predetermined start point,predetermined start position, predetermined start orientation oralignment, predetermined intermediate point(s), predeterminedintermediate position(s), predetermined intermediate orientation oralignment, predetermined end point, predetermined end position,predetermined end orientation or alignment, predetermined path,predetermined plane, predetermined cut plane, predetermined contour oroutline or cross-section or surface features or shape or projection,predetermined depth marker or depth gauge, predetermined angle ororientation or rotation marker, predetermined axis, e.g. rotation axis,flexion axis, extension axis, predetermined axis of the virtual surgicaltool, virtual surgical instrument including virtual surgical guide orcut block, virtual trial implant, virtual implant component, implant ordevice, non-visualized portions for one or more devices or implants orimplant components or surgical instruments or surgical tools, and/or oneor more of a predetermined tissue change or alteration and/or one ormore of a predetermined position and/or orientation of the virtualsurgical tool, virtual surgical instrument including virtual surgicalguide or cut block, virtual trial implant, virtual implant component,implant or device can disappear in the OHMD display when alignment isachieved again. The reverse can be possible, e.g. with the one or moreof a predetermined start point, predetermined start position,predetermined start orientation or alignment, predetermined intermediatepoint(s), predetermined intermediate position(s), predeterminedintermediate orientation or alignment, predetermined end point,predetermined end position, predetermined end orientation or alignment,predetermined path, predetermined plane, predetermined cut plane,predetermined contour or outline or cross-section or surface features orshape or projection, predetermined depth marker or depth gauge,predetermined angle or orientation or rotation marker, predeterminedaxis, e.g. rotation axis, flexion axis, extension axis, predeterminedaxis of the virtual surgical tool, virtual surgical instrument includingvirtual surgical guide or cut block, virtual trial implant, virtualimplant component, implant or device, non-visualized portions for one ormore devices or implants or implant components or surgical instrumentsor surgical tools, and/or one or more of a predetermined tissue changeor alteration and/or one or more of a predetermined position and/ororientation of the virtual surgical tool, virtual surgical instrumentincluding virtual surgical guide or cut block, virtual trial implant,virtual implant component, implant or device disappearing when alignmentis not achieved and appearing when alignment is achieved.

A magnified or minified display can be used to show one or more of apredetermined start point, predetermined start position, predeterminedstart orientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration and/or one or more of a predetermined position and/ororientation of the virtual surgical tool, virtual surgical instrumentincluding virtual surgical guide or cut block, virtual trial implant,virtual implant component, implant or device. The OHMD can also,optionally, provide or superimpose a magnified or minified display ofthe virtual anatomy or virtual data of the patient, for example afterregistration with the live anatomy/live data of the patient. Theunmagnified, magnified or minified virtual anatomy or virtual data ofthe patient can be displayed by the OHMD simultaneously, e.g. with useof different colors, grey scale or patterns, or alternatingly with theunmagnified, magnified or minified display by the OHMD of the one ormore of a predetermined start point, predetermined start position,predetermined start orientation or alignment, predetermined intermediatepoint(s), predetermined intermediate position(s), predeterminedintermediate orientation or alignment, predetermined end point,predetermined end position, predetermined end orientation or alignment,predetermined path, predetermined plane, predetermined cut plane,predetermined contour or outline or cross-section or surface features orshape or projection, predetermined depth marker or depth gauge,predetermined angle or orientation or rotation marker, predeterminedaxis, e.g. rotation axis, flexion axis, extension axis, predeterminedaxis of the virtual surgical tool, virtual surgical instrument includingvirtual surgical guide or cut block, virtual trial implant, virtualimplant component, implant or device, non-visualized portions for one ormore devices or implants or implant components or surgical instrumentsor surgical tools, and/or one or more of a predetermined tissue changeor alteration and/or one or more of a predetermined position and/ororientation of the virtual surgical tool, virtual surgical instrumentincluding virtual surgical guide or cut block, virtual trial implant,virtual implant component, implant or device. In some embodiments of theinvention, the magnification (including no magnification) orminification of the display of the virtual anatomy or virtual data ofthe patient can be the same as the magnification (including nomagnification) or minification of the one or more of a predeterminedstart point, predetermined start position, predetermined startorientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration and/or one or more of a predetermined position and/ororientation of the virtual surgical tool, virtual surgical instrumentincluding virtual surgical guide or cut block, virtual trial implant,virtual implant component, implant or device. Virtual anatomy or virtualdata of the patient as used in the foregoing includes all virtual dataof the patient, including, for example, data from vascular flow studies,metabolic imaging, kinematic data and the like.

A magnified or minified display by the OHMD can, for example, highlightif and when one or more of the surgical tool, surgical instrument, trialimplant, implant component, implant or device are not aligned with oneor more of a predetermined start point, predetermined start position,predetermined start orientation or alignment, predetermined intermediatepoint(s), predetermined intermediate position(s), predeterminedintermediate orientation or alignment, predetermined end point,predetermined end position, predetermined end orientation or alignment,predetermined path, predetermined plane, predetermined cut plane,predetermined contour or outline or cross-section or surface features orshape or projection, predetermined depth marker or depth gauge,predetermined angle or orientation or rotation marker, predeterminedaxis, e.g. rotation axis, flexion axis, extension axis, predeterminedaxis of the virtual surgical tool, virtual surgical instrument includingvirtual surgical guide or cut block, virtual trial implant, virtualimplant component, implant or device, non-visualized portions for one ormore devices or implants or implant components or surgical instrumentsor surgical tools, and/or one or more of a predetermined tissue changeor alteration and/or one or more of a predetermined position and/ororientation of the virtual surgical tool, virtual surgical instrumentincluding virtual surgical guide or cut block, virtual trial implant,virtual implant component, implant or device. In this example, thepredetermined start point, predetermined start position, predeterminedstart orientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration and/or one or more of a predetermined position and/ororientation of the virtual surgical tool, virtual surgical instrumentincluding virtual surgical guide or cut block, virtual trial implant,virtual implant component, implant or device can be magnified orminified in the OHMD display when the physical surgical tool, surgicalinstrument, trial implant, implant component, implant, and/or device arenot aligned, e.g. with the surgical plan or the one or more of thepredetermined start point, predetermined start position, predeterminedstart orientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration and/or one or more of a predetermined position and/ororientation of the virtual surgical tool, virtual surgical instrumentincluding virtual surgical guide or cut block, virtual trial implant,virtual implant component, implant or device. The one or more of apredetermined start point, predetermined start position, predeterminedstart orientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration and/or one or more of a predetermined position and/ororientation of the virtual surgical tool, virtual surgical instrumentincluding virtual surgical guide or cut block, virtual trial implant,virtual implant component, implant or device can be set to zeromagnification or minification or can go from magnified to minified orfrom minified to magnified in the OHMD display when alignment isachieved again.

If more than one a predetermined start point, predetermined startposition, predetermined start orientation or alignment, predeterminedintermediate point(s), predetermined intermediate position(s),predetermined intermediate orientation or alignment, predetermined endpoint, predetermined end position, predetermined end orientation oralignment, predetermined path, predetermined plane, predetermined cutplane, predetermined contour or outline or cross-section or surfacefeatures or shape or projection, predetermined depth marker or depthgauge, predetermined angle or orientation or rotation marker,predetermined axis, e.g. rotation axis, flexion axis, extension axis,predetermined axis of the virtual surgical tool, virtual surgicalinstrument including virtual surgical guide or cut block, virtual trialimplant, virtual implant component, implant or device, non-visualizedportions for one or more devices or implants or implant components orsurgical instruments or surgical tools, and/or one or more of apredetermined tissue change or alteration and/or one or more of apredetermined position and/or orientation of the virtual surgical tool,virtual surgical instrument including virtual surgical guide or cutblock, virtual trial implant, virtual implant component, implant ordevice are displayed by the OHMD, any combination of display styles ortechniques, e.g. multi-colored, grey scale, shaded, patterned, line,arrow, target, intermittent, appearing, disappearing, magnified,minified is possible. In some embodiments, different display styles ortechniques can be chosen for different predetermined start point,predetermined start position, predetermined start orientation oralignment, predetermined intermediate point(s), predeterminedintermediate position(s), predetermined intermediate orientation oralignment, predetermined end point, predetermined end position,predetermined end orientation or alignment, predetermined path,predetermined plane, predetermined cut plane, predetermined contour oroutline or cross-section or surface features or shape or projection,predetermined depth marker or depth gauge, predetermined angle ororientation or rotation marker, predetermined axis, e.g. rotation axis,flexion axis, extension axis, predetermined axis of the virtual surgicaltool, virtual surgical instrument including virtual surgical guide orcut block, virtual trial implant, virtual implant component, implant ordevice, non-visualized portions for one or more devices or implants orimplant components or surgical instruments or surgical tools, and/or oneor more of a predetermined tissue change or alteration and/or one ormore of a predetermined position and/or orientation of the virtualsurgical tool, virtual surgical instrument including virtual surgicalguide or cut block, virtual trial implant, virtual implant component,implant or device.

Two-Dimensional and Three-Dimensional Displays

One or more of a predetermined start point, predetermined startposition, predetermined start orientation or alignment, predeterminedintermediate point(s), predetermined intermediate position(s),predetermined intermediate orientation or alignment, predetermined endpoint, predetermined end position, predetermined end orientation oralignment, predetermined path, predetermined plane, predetermined cutplane, predetermined contour or outline or cross-section or surfacefeatures or shape or projection, predetermined depth marker or depthgauge, predetermined angle or orientation or rotation marker,predetermined axis, e.g. rotation axis, flexion axis, extension axis,predetermined axis of the virtual surgical tool, virtual surgicalinstrument including virtual surgical guide or cut block, virtual trialimplant, virtual implant component, implant or device, non-visualizedportions for one or more devices or implants or implant components orsurgical instruments or surgical tools, and/or one or more of apredetermined tissue change or alteration and/or one or more of apredetermined position and/or orientation of the virtual surgical tool,virtual surgical instrument including virtual surgical guide or cutblock, virtual trial implant, virtual implant component, implant ordevice can be displayed by the OHMD in two dimensions.

One or more of a predetermined start point, predetermined startposition, predetermined start orientation or alignment, predeterminedintermediate point(s), predetermined intermediate position(s),predetermined intermediate orientation or alignment, predetermined endpoint, predetermined end position, predetermined end orientation oralignment, predetermined path, predetermined plane, predetermined cutplane, predetermined contour or outline or cross-section or surfacefeatures or shape or projection, predetermined depth marker or depthgauge, predetermined angle or orientation or rotation marker,predetermined axis, e.g. rotation axis, flexion axis, extension axis,predetermined axis of the virtual surgical tool, virtual surgicalinstrument including virtual surgical guide or cut block, virtual trialimplant, virtual implant component, implant or device, non-visualizedportions for one or more devices or implants or implant components orsurgical instruments or surgical tools, and/or one or more of apredetermined tissue change or alteration and/or one or more of apredetermined position and/or orientation of the virtual surgical tool,virtual surgical instrument including virtual surgical guide or cutblock, virtual trial implant, virtual implant component, implant ordevice can be displayed by the OHMD in three dimensions.

One or more of a predetermined start point, predetermined startposition, predetermined start orientation or alignment, predeterminedintermediate point(s), predetermined intermediate position(s),predetermined intermediate orientation or alignment, predetermined endpoint, predetermined end position, predetermined end orientation oralignment, predetermined path, predetermined plane, predetermined cutplane, predetermined contour or outline or cross-section or surfacefeatures or shape or projection, predetermined depth marker or depthgauge, predetermined angle or orientation or rotation marker,predetermined axis, e.g. rotation axis, flexion axis, extension axis,predetermined axis of the virtual surgical tool, virtual surgicalinstrument including virtual surgical guide or cut block, virtual trialimplant, virtual implant component, implant or device, non-visualizedportions for one or more devices or implants or implant components orsurgical instruments or surgical tools, and/or one or more of apredetermined tissue change or alteration and/or one or more of apredetermined position and/or orientation of the virtual surgical tool,virtual surgical instrument including virtual surgical guide or cutblock, virtual trial implant, virtual implant component, implant ordevice can be displayed by the OHMD in two dimensions and/or threedimensions, for example alternatingly or as triggered by voice commandsor other commands. Simultaneous display of one or more of apredetermined start point, predetermined start position, predeterminedstart orientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration and/or one or more of a predetermined position and/ororientation of the virtual surgical tool, virtual surgical instrumentincluding virtual surgical guide or cut block, virtual trial implant,virtual implant component, implant or device in three dimensions can bepossible.

Stereoscopic and Non-Stereoscopic Displays

One or more of a predetermined start point, predetermined startposition, predetermined start orientation or alignment, predeterminedintermediate point(s), predetermined intermediate position(s),predetermined intermediate orientation or alignment, predetermined endpoint, predetermined end position, predetermined end orientation oralignment, predetermined path, predetermined plane, predetermined cutplane, predetermined contour or outline or cross-section or surfacefeatures or shape or projection, predetermined depth marker or depthgauge, predetermined angle or orientation or rotation marker,predetermined axis, e.g. rotation axis, flexion axis, extension axis,predetermined axis of the virtual surgical tool, virtual surgicalinstrument including virtual surgical guide or cut block, virtual trialimplant, virtual implant component, implant or device, non-visualizedportions for one or more devices or implants or implant components orsurgical instruments or surgical tools, and/or one or more of apredetermined tissue change or alteration and/or one or more of apredetermined position and/or orientation of the virtual surgical tool,virtual surgical instrument including virtual surgical guide or cutblock, virtual trial implant, virtual implant component, implant ordevice can be displayed by the OHMD in a non-stereoscopic manner inthree dimensions, with similar view angle of the virtual data of thepatient seen by the surgeon's eyes through the display of the OHMD unitand the live data of the patient seen by the surgeon's eyes through theOHMD unit.

One or more of a predetermined start point, predetermined startposition, predetermined start orientation or alignment, predeterminedintermediate point(s), predetermined intermediate position(s),predetermined intermediate orientation or alignment, predetermined endpoint, predetermined end position, predetermined end orientation oralignment, predetermined path, predetermined plane, predetermined cutplane, predetermined contour or outline or cross-section or surfacefeatures or shape or projection, predetermined depth marker or depthgauge, predetermined angle or orientation or rotation marker,predetermined axis, e.g. rotation axis, flexion axis, extension axis,predetermined axis of the virtual surgical tool, virtual surgicalinstrument including virtual surgical guide or cut block, virtual trialimplant, virtual implant component, implant or device, non-visualizedportions for one or more devices or implants or implant components orsurgical instruments or surgical tools, and/or one or more of apredetermined tissue change or alteration and/or one or more of apredetermined position and/or orientation of the virtual surgical tool,virtual surgical instrument including virtual surgical guide or cutblock, virtual trial implant, virtual implant component, implant ordevice can be displayed by the OHMD in a stereoscopic manner in threedimensions.

One or more of a predetermined start point, predetermined startposition, predetermined start orientation or alignment, predeterminedintermediate point(s), predetermined intermediate position(s),predetermined intermediate orientation or alignment, predetermined endpoint, predetermined end position, predetermined end orientation oralignment, predetermined path, predetermined plane, predetermined cutplane, predetermined contour or outline or cross-section or surfacefeatures or shape or projection, predetermined depth marker or depthgauge, predetermined angle or orientation or rotation marker,predetermined axis, e.g. rotation axis, flexion axis, extension axis,predetermined axis of the virtual surgical tool, virtual surgicalinstrument including virtual surgical guide or cut block, virtual trialimplant, virtual implant component, implant or device, non-visualizedportions for one or more devices or implants or implant components orsurgical instruments or surgical tools, and/or one or more of apredetermined tissue change or alteration and/or one or more of apredetermined position and/or orientation of the virtual surgical tool,virtual surgical instrument including virtual surgical guide or cutblock, virtual trial implant, virtual implant component, implant ordevice can be displayed by the OHMD in a stereoscopic and/or anon-stereoscopic display, for example alternatingly or as triggered byvoice commands or other commands. Simultaneous display of one or more ofa predetermined start point, predetermined start position, predeterminedstart orientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration and/or one or more of a predetermined position and/ororientation of the virtual surgical tool, virtual surgical instrumentincluding virtual surgical guide or cut block, virtual trial implant,virtual implant component, implant or device in a non-stereoscopicmanner with display of one or more of a predetermined start point,predetermined start position, predetermined start orientation oralignment, predetermined intermediate point(s), predeterminedintermediate position(s), predetermined intermediate orientation oralignment, predetermined end point, predetermined end position,predetermined end orientation or alignment, predetermined path,predetermined plane, predetermined cut plane, predetermined contour oroutline or cross-section or surface features or shape or projection,predetermined depth marker or depth gauge, predetermined angle ororientation or rotation marker, predetermined axis, e.g. rotation axis,flexion axis, extension axis, predetermined axis of the virtual surgicaltool, virtual surgical instrument including virtual surgical guide orcut block, virtual trial implant, virtual implant component, implant ordevice, non-visualized portions for one or more devices or implants orimplant components or surgical instruments or surgical tools, and/or oneor more of a predetermined tissue change or alteration and/or one ormore of a predetermined position and/or orientation of the virtualsurgical tool, virtual surgical instrument including virtual surgicalguide or cut block, virtual trial implant, virtual implant component,implant or device in a stereoscopic manner can be possible.

In some embodiments of the invention, one or more of a predeterminedstart point, predetermined start position, predetermined startorientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration and/or one or more of a predetermined position and/ororientation of the virtual surgical tool, virtual surgical instrumentincluding virtual surgical guide or cut block, virtual trial implant,virtual implant component, implant or device can be located in a spine,more specifically a vertebral body, a pedicle, a vertebral fracture, aposterior element, a facet joint depending on the virtual surgical planand the anatomy and clinical condition of the patient. The predeterminedstart point, predetermined start position, predetermined startorientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration and/or one or more of a predetermined position and/ororientation of the virtual surgical tool, virtual surgical instrumentincluding virtual surgical guide or cut block, virtual trial implant,virtual implant component, implant or device can be located in theposterior elements of a spine, a pedicle and a vertebral body, forexample, if spinal fusion with pedicle screws or vertebroplasty ofkyphoplasty are contemplated.

If spinal fusion with pedicle screws is planned, the predetermined startpoint, predetermined start position, predetermined start orientation oralignment, predetermined intermediate point(s), predeterminedintermediate position(s), predetermined intermediate orientation oralignment, predetermined end point, predetermined end position,predetermined end orientation or alignment, predetermined path,predetermined plane, predetermined cut plane, predetermined contour oroutline or cross-section or surface features or shape or projection,predetermined depth marker or depth gauge, predetermined angle ororientation or rotation marker, predetermined axis, e.g. rotation axis,flexion axis, extension axis, predetermined axis of the virtual surgicaltool, virtual surgical instrument including virtual surgical guide orcut block, virtual trial implant, virtual implant component, implant ordevice, non-visualized portions for one or more devices or implants orimplant components or surgical instruments or surgical tools, and/or oneor more of a predetermined tissue change or alteration and/or one ormore of a predetermined position and/or orientation of the virtualsurgical tool, virtual surgical instrument including virtual surgicalguide or cut block, virtual trial implant, virtual implant component,implant or device can coincide with, be parallel with, or be alignedand/or superimposed with the long axis of the pedicle screw in itsintended virtual placement position from the virtual surgical plan,optionally using placement criteria, e.g. distance from cortex, as usedin the virtual surgical plan.

If vertebroplasty or kyphoplasty or spinal biopsy is planned, thepredetermined start point, predetermined start position, predeterminedstart orientation or alignment, predetermined intermediate point(s),predetermined intermediate position(s), predetermined intermediateorientation or alignment, predetermined end point, predetermined endposition, predetermined end orientation or alignment, predeterminedpath, predetermined plane, predetermined cut plane, predeterminedcontour or outline or cross-section or surface features or shape orprojection, predetermined depth marker or depth gauge, predeterminedangle or orientation or rotation marker, predetermined axis, e.g.rotation axis, flexion axis, extension axis, predetermined axis of thevirtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, non-visualized portions for one or moredevices or implants or implant components or surgical instruments orsurgical tools, and/or one or more of a predetermined tissue change oralteration and/or one or more of a predetermined position and/ororientation of the virtual surgical tool, virtual surgical instrumentincluding virtual surgical guide or cut block, virtual trial implant,virtual implant component, implant or device can coincide with, beparallel with, or be aligned and/or superimposed with the long axis ofthe vertebroplasty, kyphoplasty or biopsy needle or needle set in itsintended virtual placement position from the virtual surgical plan,optionally using placement criteria, e.g. distance from cortex, as usedin the virtual surgical plan.

When stereoscopic projection is used by the OHMD, the display for theleft eye and the right eye can be adjusted for the surgeon's oroperator's inter-ocular distance, including, for example, theinter-pupillary distance. For example, the distance between the leftpupil and the right pupil can be measured prior to operating the OHMD.Such measurements can be performed using an image and/or video capturesystem integrated into, attached to or separate from the OHMD. Suchmeasurements can also be performed using any other technique known inthe art, including, for example, mechanical rulers, optical measurementtools and standard tools used by optometrists.

Adjusting the OHMD Unit Including the Display

In some embodiments, once the inter-ocular, e.g. the inter-pupillarydistance, of the surgeon or operator is known, it can be entered intothe display system interface and/or software and the 3D projection ofthe left and the right eye can be adjusted for the user. For example,with a narrow inter-ocular or inter-pupillary distance, the projectionfor the left eye and the right eye can be moved closer to the nose sothat the center of the left and the right projections will be alignedwith the center of the left eye/pupil and the right eye/pupil. With awide inter-ocular or inter-pupillary distance, the projection for theleft eye and the right eye can be moved further away from the nose sothat the center of the left and the right projections will be alignedwith the center of the left eye/pupil and the right eye/pupil. Differentuser settings can be stored in the system, e.g. by user name. In thismanner, when a different user is placing the OHMD on his or her head,the user or the system can call up their preferred user settings,including their respective inter-ocular or inter-pupillary distance.User settings can be called up, for example, using a visual or opticalkeyboard interface, projected by the OHMD, where the operator can selectvirtual buttons. User settings can also be called up using voicecommands, keyboards and any other known technique or technique forexecuting user commands.

Refresh Rates, Addressing Image Flicker

In many embodiments of the invention, a fast refresh rate can bedesirable, e.g. 15 Hz, 20 Hz, 25 Hz, or 30 Hz, 50 Hz, 70 Hz, 80 Hz, 100Hz, 120 Hz, 150 Hz, 175 Hz, 200 Hz or greater. When higher refresh ratesare used, the spatial resolution of the display of the virtual data canoptionally be reduced if bandwidth and transmission speed and/or displayspeed reach their limits. Alternatively, there can be an alternating ofa high resolution display, e.g. 1920×1080 pixel resolution, and lowerresolution, e.g. 1024×768 pixel resolution. The ratio of high to lowerresolution images can be 1:1, 2:1, 3:1, 1:2, 1:3, with any othercombination possible.

Some users physicalize no flicker with refresh rates of 30 Hz, sometimesless. Other users can feel or experience flicker with refresh rates of70 Hz or faster. If a user is experiencing flicker effects or a flickerfeeling with the display of virtual data, the user can have the optionof increasing the refresh rate and, optionally, decreasing the displayresolution if necessary, for example for reasons of bandwidth ortransmission speed. The user can also select alternating resolutions,e.g. 1920×1080 pixel resolution intermixed with 1024×768 pixelresolution; any other pixel resolution and combination of pixelresolutions is possible. In this manner, the user can select the settingthat will yield a pleasant, substantially flicker free display while atthe same time maintaining sufficient spatial and/or temporal resolutionto enable an accurate physical/virtual work environment.

In some embodiments of the invention, the display will automaticallyturn of and, optionally, turn on depending where the user and/oroperator and/or surgeon directs the view.

Automated Turning Off and/or Turning On

In select circumstances, the user and/or operator and/or surgeon mayelect to turn off the OHMD display or to turn it back on. The turningoff and/or on can be executed via voice commands. It can also beexecuted via gesture commands, eye commands, digital/finger commands ona physical or virtual keypad or keyboard, e.g. projected by the OHMD.

In some embodiments, the OHMD display can turn off and/or onautomatically. The turning off and/or on can be triggered by any numberof initiating events or movements, which can optionally be defined bythe user. Events or movements triggering an automatic turning off and/orturning on can be different between different users and can be stored asuser preferences.

Automatic turning off and/or turning on can, for example, also helpreduce the times the OHMD display is on or active, which can bedesirable when users experience a flicker like feeling or encounter aflicker experience with the OHMD display or other feelings ofdiscomfort. In this way, the periods of potential flicker exposure orother feelings of discomfort can be reduced to the key parts or portionsor sections when the user requires the OHMD to execute an activity, e.g.a physical surgical step optionally defined in a virtual surgical planwith display of one or more of a predetermined start point,predetermined start position, predetermined start orientation oralignment, predetermined intermediate point(s), predeterminedintermediate position(s), predetermined intermediate orientation oralignment, predetermined end point, predetermined end position,predetermined end orientation or alignment, predetermined path,predetermined plane, predetermined cut plane, predetermined contour oroutline or cross-section or surface features or shape or projection,predetermined depth marker or depth gauge, predetermined angle ororientation or rotation marker, predetermined axis, e.g. rotation axis,flexion axis, extension axis, predetermined axis of the virtual surgicaltool, virtual surgical instrument including virtual surgical guide orcut block, virtual trial implant, virtual implant component, implant ordevice, non-visualized portions for one or more devices or implants orimplant components or surgical instruments or surgical tools, and/or oneor more of a predetermined tissue change or alteration and/or one ormore of a predetermined position and/or orientation of the virtualsurgical tool, virtual surgical instrument including virtual surgicalguide or cut block, virtual trial implant, virtual implant component,implant or device.

In some embodiments of the invention, the OHMD display can optionallyautomatically turn on when the user looks at the target area ofactivity, e.g. a surgical field or an organ or a tissue located withinthe coordinates of a live surgical field or area and/or a virtualsurgical field or area. In some embodiments of the invention, the OHMDdisplay can optionally automatically turn off when the user looks awayfrom the target area of activity, e.g. a surgical field or area or anorgan or a tissue located within the coordinates of a live and/orvirtual surgical field or area. In some embodiments, the OHMD displaycan optionally automatically turn on when the user looks at the targetarea of activity, e.g. a surgical field or an organ or a tissue locatedwithin the coordinates of a live surgical field or area and/or a virtualsurgical field or area, and one or more optical markers are detected inthe surgical field or an organ or a tissue located within thecoordinates of the live surgical field or area and/or a virtual surgicalfield or area by a camera, image or video capture system integratedinto, attached to or separate from the OHMD. In some embodiments of theinvention, the OHMD display can optionally automatically turn off whenthe user looks away from the target area of activity, e.g. a surgicalfield or area or an organ or a tissue located within the coordinates ofa live and/or virtual surgical field or area, and one or more opticalmarkers are absent from the surgeon's view and/or the view of a camera,an image or video capture system integrated into, attached to orseparate from the OHMD.

The target area of activity, e.g. a surgical field and/or a targettissue can be defined and/or identified using different means, e.g.image capture, optical markers, navigation markers including infraredmarkers, retroreflective markers, RF markers, surgical navigation, LEDs,reference phantoms, calibration phantoms, marks drawn on the targetarea, e.g. on the skin or a surgical drape. If surgery is contemplated,any of the foregoing active and/or passive markers can be placed on thepatient, e.g. underneath a surgical drape, or within the visiblesterile, exposed area of the patient on which the surgery will beperformed. Alternatively, any active or passive markers can also beplaced on top of the sterile drape or on the patient's skin, e.g.surrounding the surgical area or surgical field. A target area can alsobe identified with use of one or more anatomic landmarks, e.g. in a hipa most inferior point, e.g. sulcus point, between the greater trochanterand the femoral neck, a most superior point on the greater trochanter, amost superior point on a lesser trochanter, an acetabular rim, anacetabular center or in a knee a most medial point on a medial condyle,a most lateral point on a lateral condyle, a center of a trochlearnotch, a tibial spine, a most anterior point of a tibia, a central pointof a patella. One or more of the same landmarks that have been/are beingused for registration of virtual data and live data of the patient canbe used for defining or identifying a target area of activity. Thelandmarks can be identified using, for example, an image and/or videocapture system integrated into, attached to or separate from an OHMD.The landmarks can be identified by attaching optionally one or moreoptical markers, navigation markers including infrared markers,retroreflective markers, RF markers, surgical navigation, LEDs,reference phantoms, calibration phantoms, or marks. A target area can beenclosed by landmarks, e.g. by three or more landmarks. A target areacan extend beyond one or more landmarks, e.g. by 2, 4, 5, 6, 8, 10 cm ormore or any other distance or radius, e.g. selected by the surgeon oroperator.

If image capture is used to define an area of intended activity, e.g. asurgical field, the user and/or the surgeon can optionally look at thearea of intended activity, e.g. the intended field. Optionally, identifythe center area of the area of activity and/or the surgical field can bedefined by the user, e.g. by pointing at it with a finger or a pointingdevice or an RF marker, an optical marker, a navigation marker includinginfrared markers, retroreflective markers, RF markers, an LED and/or acalibration phantom or a reference phantom. Once the user's and/or thesurgeon's view is focused on the intended area of activity and/or theintended surgical field, the user and/or surgeon can execute a command,e.g. a voice command or a finger command, to identify the intended areaof activity and/or the surgical field and to store it in the imageand/or video capture system. The identified intended area of activityand/or the surgical field is in this manner memorized in the imageand/or video capture system. Using standard image processing techniques,the image and/or video capture system can subsequently identify if 20%,30%, 40%, 50%, 60%, 70%, 80%, 90% or 100% of the intended area ofactivity and/or the surgical field are included in the field of view ofthe user and/or the operator and/or the surgeon. Once a certainpercentage, e.g. 50% or 60% or 70% or 80% or 90% of the area of intendedactivity and/or the surgical field is included in the field of view ofthe surgeon, the OHMD can automatically turn on the OHMD display.Optionally, as the user, operator and/or surgeon turns away his or herview, the OHMD display can automatically turn off, e.g. when less than90%, 80%, 70%, 60% or 50% of the intended area of activity and/orsurgical field area included in the field of view.

The area or percentage used for turning on the OHMD and for turning offthe OHMD can be different. The percentage can be selected and,optionally, stored as a user preference.

The field of view can be defined in various different ways, optionallyas a user preference. For example, the field of view can be the areacovered by the OHMD display when the user is looking through the OHMD.The field of view can be the entire visual field available to the userand/or operator and/or surgeon. The field of view can be a subsection ofthe visual field of the user, operator and/or surgeon.

Rather than using a percentage of area of the intended area of activityand/or surgical field included, other triggers can be used using, forexample, anatomic landmarks, image capture or optical markers, anavigation marker including infrared markers, retroreflective markers,RF markers, an LED and/or a calibration phantom or a reference phantom.For example, the OHMD display can automatically turn on when the user,operator and/or surgeon starts looking at the intended area of activityand/or the surgical field when an anatomic landmark, an optical marker,a navigation marker including infrared markers, retroreflective markers,RF markers, an LED or a calibration phantom or reference phantom (e.g.as seen through an image and/or video capture system integrated into,attached to or separate from the OHMD) is located in the outer onethird, central one third or inner one third or the inner half or otherlandmark or demarcation/separation of the field of view.

Alternatively, the OHMD display can also automatically turn on when thefield of view reaches within a certain centimeter range of one or moreof an anatomic landmark, an optical marker, a navigation markerincluding infrared markers, retroreflective markers, RF markers, an LEDor a calibration phantom or reference phantom or IMU, e.g. within 15 cm,10 cm, 5 cm etc. The OHMD display can also automatically turn on whenthe field of view reaches within a certain centimeter range of one ormore of a marker of an area of intended activity and/or a surgicalfield, e.g. a pin or a screw, e.g. within 15 cm, 10 cm, 5 cm etc.

The OHMD display can optionally also automatically turn off when theintended area of activity or the surgical field or area decreases belowa certain threshold percentage (optionally set by the user) of the fieldof view, e.g. 90%, 80%, 70%, 60%, 50%, 40%, 30%, 20%, 10%, etc. In someembodiments, the OHMD display can automatically turn off when the user,operator and/or surgeon starts looking away from the intended area ofactivity and/or the surgical field when an anatomic landmark, an opticalmarker, a navigation marker including infrared markers, retroreflectivemarkers, RF markers, an LED or a calibration phantom or referencephantom (e.g. as seen through an image and/or video capture systemintegrated into, attached to or separate from the OHMD) is locatedoutside the outer one third, central one third or inner one third or theinner half or other landmark or demarcation/separation of the field ofview. Alternatively, the OHMD display can also automatically turn offwhen the field of view reaches outside a certain centimeter range of oneor more of an anatomic landmark, an optical marker, a navigation markerincluding infrared markers, retroreflective markers, RF markers, an LEDor a calibration phantom or reference phantom or IMU, e.g. outside 5 cm,10 cm, 15 cm or more. The OHMD display can also automatically turn offwhen the field of view reaches outside a certain centimeter range of oneor more of a marker of an area of intended activity and/or a surgicalfield, e.g. a pin or a screw, e.g. within 5 cm, 10 cm, 15 cm or more.

In some embodiments of the invention, the OHMD display can automaticallyturn on when a select surgical instrument, e.g. an awl or a pin driveror a reamer or a saw, or a select medical device component, or multiplethereof (either simultaneously or sequentially) appear in the field ofview. Optionally, the OHMD display can automatically turn off when aselect surgical instrument, e.g. an awl or a pin driver or a reamer or asaw, or a select medical device component, or multiple thereof (eithersimultaneously or sequentially) disappear from the field of view. Theappearing or disappearing of the one or more surgical instruments ormedical device components can be caused by the user/surgeon moving thehead away from the intended area of activity and/or the surgical field;it can also be caused by the user/surgeon moving the surgical instrumentand/or medical device component outside the field of view or away fromthe area of intended activity and/or surgical field while continuing tolook at the area of intended activity and/or the surgical field.

In some embodiments of the invention, the OHMD display can automaticallyturn off when the user/operator/surgeon looks at a display monitor otherthan the OHMD. Such a display monitor can be a video screen or a TVscreen or a computer monitor, or a PACS monitor or other displaymonitor, e.g. located in an operating room or in a factory. In someembodiments of the invention, the monitor can be recognized, e.g. basedon its square or rectangular outline/shape using image capture combinedwith standard image processing techniques. In some embodiments of theinvention, a video screen or a TV screen or a computer monitor, or aPACS monitor or other display monitor can be identified for example withoptical markers, navigation markers including infrared markers,retroreflective markers, RF markers, LEDs and other markers placed on,surrounding or nearby the monitor.

Optionally, the OHMD can automatically turn back on when theuser/operator/surgeon looks away from the monitor. The turning on andturning off of the OHMD display can be triggered, for example, when themonitor occupies 25%, 50% or 75% or more of the field of view. Theturning on and turning off of the OHMD display can be triggered, forexample, when the field of view, e.g. through the OHMD display, reachesthe central area of the point of the monitor, or within 5 cm, 10 cm, 15cm or more of the central area or point of the monitor, or within acertain distance or area from an optical marker, navigation markerincluding infrared markers, retroreflective markers, RF markers, LEDand/or other markers placed on, surrounding or nearby the monitor.

In select embodiments, it can be preferable that the OHMD display turnson when the user looks at the monitor and it turns off when the userlooks away from the monitor. It can then optionally turn back on whenthe user looks at the intended area of activity, e.g. a surgical fieldor area.

In some embodiments of the invention, when flicker or the feeling ofexperiencing flicker with the OHMD display is a concern for a user, theOHMD can turn on an off on an intermittent basis, e.g. it can displaythe virtual data for 1, 2, 3, 4 or more seconds and then turn off for abreak, e.g. 1, 2, 3, 4 or more seconds. The periods of display on anddisplay off can be defined by the user based on user preferences. Theperiods of display on and off can be combined with various triggers ofautomatic turning on and off of the OHMD display, as outlined, forexample in the foregoing.

The foregoing embodiments and examples describing non-automated andautomated or automatic techniques for turning on an OHMD display andturning off an OHMD display are only exemplary in nature and are in noway meant to be limiting of the invention. Someone skilled in the artcan recognize many different triggers for turning on and off an OHMDdisplay automatically. The automatic turning on and off of an OHMDdisplay can also be a useful feature for preserving battery life, e.g.disposable or rechargeable.

Managing Display, Hardware, Software or Bandwidth Limitations

In some embodiments of the invention, the display of the OHMD unit candisplay a subset of the data and/or images representing a smallerportion of the field of view visible through the OHMD or displayable bythe display of the OHMD unit, using, for example, only a portion of theavailable display. If data from a pre-operative or intra-operativeimaging study, e.g. x-rays, a CT scan, an MRI scan, are displayed, thedata or images displayed by the OHMD can also be targeted to a volumesmaller than the original scan volume or area covered by the imagingstudy in order to decrease the amount of data displayed. In addition,the data or images displayed by the OHMD can also be targeted to avolume or area smaller than the volume or area to be operated or smallerthan the volume or area of the surgical site. This embodiment can, forexample, be useful, when the software environment limits the amount ofsurface points or nodes displayed or limits the size or amount of thedata displayed by the OHMD. This embodiment can also be useful when aWiFi or Bluetooth or other wireless connection is used with the OHMDwith limitations in bandwidth and/or data transmission, thereby limitingthe amount of data being transmitted to the OHMD and, ultimately,displayed, in particular when this limitation implies a limitation inthe amount of data available for the display of the data and/or imagesby the OHMD.

This smaller portion of the field of view visible through the OHMD ordisplayable by the display of the OHMD unit, smaller, targeted volumefrom an imaging study, or the volume or area smaller that the volume orarea of the surgical site can be targeted to portions of the surgicalsite or to anatomic landmarks. For example, in a knee replacement, thissmaller portion of the field of view can be targeted to the distal femuror portions of the distal femur while the surgeon is contemplatingsurgical steps on the femur, e.g. a distal femoral cut or an anterior orposterior cut or chamfer cuts; it can be targeted to the proximal tibiaor portions thereof while the surgeon is contemplating surgical steps onthe tibia, e.g. a proximal tibial cut or a tibial keel preparation andpunch; it can be targeted to the patella, while the surgeon iscontemplating surgical steps on the patella, e.g. a milling or cuttingof the patella. In a hip replacement, the smaller portion of the fieldof view can be targeted to the proximal femur or portions thereof, whilethe surgeon is contemplating steps on the proximal femur, e.g. a femoralneck cut; it can be targeted to the acetabulum, while the surgeon iscontemplating surgical steps on the acetabulum, e.g. an acetabularreaming or an impaction of an acetabular cup; it can be re-focused orre-targeted on the proximal femur when the surgeon contemplates femoralbroaching or reaming, optionally followed by femoral componentimpaction. In a pedicle screw placement or a vertebroplasty orkyphoplasty, the smaller portion of the field of view can be targeted tothe level and/or the side where the surgeon contemplates the nextsurgical step, e.g. an insertion of an awl, a pedicle screw, a needle, avertebra- or kyphoplasty needle.

A targeted area or smaller portion of the field of view visible throughthe OHMD or displayable by the display of the OHMD, a smaller, targetedvolume from an imaging study, or a volume or area smaller that thevolume or area of the surgical site can also be defined with use of oneor more anatomic landmarks, e.g. in a hip a most inferior point, e.g.sulcus point, between the greater trochanter and the femoral neck, amost superior point on the greater trochanter, a most superior point ona lesser trochanter, an acetabular rim or portions thereof, anacetabular center, or in a knee, a most medial point on a medialcondyle, a most lateral point on a lateral condyle, a center of atrochlear notch, a tibial spine, a most anterior point of a tibia, acentral point of a patella. One or more of the same landmarks that havebeen/are being used for registration of virtual data and live data ofthe patient can be used for defining or identifying a target area or asmaller portion of the field of view visible through the OHMD ordisplayable by the display of the OHMD. The landmarks can be identifiedusing, for example, an image and/or video capture system integratedinto, attached to or separate from an OHMD. The landmarks can beidentified by attaching optionally one or more optical markers,navigation markers including infrared markers, retroreflective markers,RF markers, surgical navigation, LEDs, reference phantoms, calibrationphantoms, or marks. A target area can be enclosed by landmarks, e.g. bythree or more landmarks. A target area can extend beyond one or morelandmarks, e.g. by 2, 4, 5, 6, 8, 10 cm or more or any other distance orradius, e.g. selected by the surgeon or operator.

By limiting the display to such a smaller portion of the field of viewvisible through the OHMD or displayable by the display of the OHMD ortarget area, a smaller, targeted volume from an imaging study, or avolume or area smaller that the volume or area of the surgical site theamount of data displayed can be reduced. In addition, the amount of datatransmitted, e.g. using a Wifi, Bluetooth or LiF network can also bereduced.

Viewing 2D Computer Monitors Through an OHMD Unit

In some embodiments of the invention, the OHMD system can detect, e.g.automatically, if the surgeon or operator is looking at a computer ordisplay monitor separate from the OHMD, for example, with use of animage and/or video capture system integrated into, attached to orseparate from the OHMD. The standalone or separate computer or displaymonitor can be used, for example, to display image data, e.g. of apatient, or to concurrently display virtual data displayed by the OHMD.The image and/or video capture system can, for example, capture theoutline of the computer or display monitor, e.g. round, square orrectangular, and the software can, optionally, automatically match,superimpose or align the items or structures displayed by the OHMD withthe items or structures displayed by the standalone or separate computeror display monitor. Alternatively, the user, operator and/or surgeon canexecute a command, e.g. a voice command or a command using a virtualfinger/keyboard interface, indicating that he or she is looking at thestandalone or separate computer or display monitor and the software canthen match, superimpose or align the items or structures displayed bythe OHMD with the items or structures displayed by the standalone orseparate computer or display monitor. The OHMD system can match,superimpose, or align all of the structures displayed by the standaloneor separate computer monitor. The OHMD system can match, superimpose oralign a portion of the structures displayed by the standalone orseparate computer monitor.

The OHMD can display the structures displayed by the standalone orseparate computer monitor using the same color. The OHMD can display thestructures displayed by the standalone or separate computer monitorusing the different colors. The OHMD can display structures notdisplayed by the standalone or separate computer monitor using adifferent color or greyscale or contrast than that used by thestandalone or separate computer monitor. The OHMD can display thestructures displayed by the standalone or separate computer monitorusing the same greyscale and/or contrast used by the standalone orseparate computer monitor. The OHMD can display the structures displayedby the standalone or separate computer monitor using a differentgreyscale and/or contrast used by the standalone or separate computermonitor.

The OHMD can display the structures displayed by the standalone orseparate computer monitor using the same image intensity used by thestandalone or separate computer monitor. The OHMD can display thestructures displayed by the standalone or separate computer monitorusing a different image intensity used by the standalone or separatecomputer monitor, e.g. brighter or less bright.

In some embodiments of the invention, a standalone or separate computeror display monitor located in a user area, e.g. an operating room or asurgical suite, can be used as a calibration or reference orregistration phantom for the OHMD unit including the frame and displayposition, orientation and/or alignment and/or direction of movement. Themonitor can have a round, rectangular or square shape of knowndimensions. An image and/or video capture system integrated into,attached to or separate from the OHMD can be used to capture one or moreimages of the monitor. Since the dimensions of the monitor are known,the size, shape or dimensions, for example along its edges, or the areaof the monitor on the captured image(s) can be used to determine thedistance of the OHMD to the monitor; the shape of the circle, oval,rectangle or square can be used to determine the angle of the OHMDrelative to the monitor. If the image and/or video capture systemintegrated into or attached to the OHMD uses two or more cameras, thedifference in shape of the circle, oval, rectangle or square detectedbetween a first, second and any additional cameras can be used toincrease the accuracy of any estimates of the angular orientation of theOHMD to the display monitor, e.g. by calibrating the measurement of afirst camera against a second camera against a third camera and soforth. If two or more cameras are used integrated into or attached todifferent portions of the OHMD frame, e.g. the left side of the frameand the right side of the frame, the difference in projection of themonitor circle, oval, rectangle or square between the two cameras canalso be used to estimate the user's head position and/or orientationand/or alignment and/or the position and/or orientation and/or alignmentof the OHMD frame in relationship to the user's head and/or face.

In some embodiments of the invention, the user and/or surgeon canoptionally look at the display monitor through the OHMD whilemaintaining his or her head in a neutral position, e.g. with no neckabduction, adduction, flexion, extension or rotation. This head positioncan be used to calibrate the position of the OHMD display inrelationship to the target area and/or the patient and/or the surgicalsite, e.g. during an initial registration or a subsequent registration.This head position can also be used to calibrate the position of theOHMD unit/frame in relationship to the user's and/or the surgeon's headand face. Optionally, the user and/or surgeon can place his or her headon a chin stand or head holder for purposes of this calibration orregistration. This process of using an external computer or displaymonitor as a reference for calibration and/or registration purposes canbe performed at the beginning of an activity and/or a surgicalprocedure, e.g. as part of an initial registration process. This processof using an external display monitor as a reference for calibrationand/or registration purposes can also be performed during an activity orafter an activity and/or surgical procedure, for example when there isconcern that the OHMD unit may have moved relative to the user's and/orsurgeon's face.

In some embodiments of the invention, the position, location,orientation, and/or alignment of the outline of the standalone orseparate computer or display monitor can be monitored, for example usingan image and/or video capture system integrated into, attached to orseparate from the OHMD. Optionally, the position, location, orientationand/or alignment of the outline of the standalone or separate computeror display monitor can be monitored using attached optical markers,navigation markers including infrared markers, retroreflective markers,RF markers, LEDs and/or IMUs as well as any other techniques describedin the specification or known in the art for determining and/or trackingthe position, location, orientation and/or alignment of an object. Withthe position, location, orientation and/or alignment of the standaloneor external computer or display monitor known, the position, location,orientation, alignment and/or direction of movement of the OHMD unit canbe tracked in relationship to it, e.g. via an image and/or video capturesystem integrated into or attached to the OHMD or optical markers,navigation markers including infrared markers, retroreflective markers,RF markers, LEDs and/or IMUs integrated into it or attached to it. Asthe position, location, orientation, alignment and/or direction ofmovement of the OHMD unit can be tracked, the display of the OHMD unitcan at all times or, if preferred, intermittently, display the samestructures, or at least a portion or subset thereof, displayed by thestandalone or separate computer or display monitor, spatially matched.If the standalone or separate computer or display monitor occupies onlya portion of the visual field covered by the OHMD display, the OHMDdisplay can match the displayed structures with the structures displayedby the standalone or separate computer or display monitor only for theportion of the visual field occupied by the standalone or separatecomputer or display monitor. Optionally, the OHMD display can displaystructures extending beyond the portion of the visual field occupied bythe standalone or separate computer or display monitor. The structuresextending beyond the portion of the visual field occupied by thestandalone or separate computer or display monitor can be continuouswith the structures displayed by the standalone or separate computer ordisplay monitor. The structures outside the portion of the visual fieldoccupied by the standalone or separate computer or display monitor canbe separate and/or from the structures displayed by the standalone orseparate computer or display monitor. For example, in addition todisplaying one or more structures matching or corresponding to what isdisplayed by the standalone or separate computer or display monitor, theOHMD display can display items such as vital signs or patientdemographics, or pre-operative imaging studies in those portions of thevisual field that do not include the standalone or separate computer ordisplay monitor. This can be useful when the user, operator and/orsurgeon is not looking at the patient.

In some embodiments of the invention, the OHMD can display surgicalfield related information, e.g. details or aspects of a virtual surgicalplan, e.g. intended/projected cut planes, or anatomic information of thepatient, e.g. from a pre-operative imaging study, when the user orsurgeon is looking at the surgical field; the OHMD can display portionsof information or all of the information displayed by a standalone orseparate computer or display monitor, for example in 3D while thestandalone or separate computer or display monitor display can be in 2D,when the user or surgeon is looking at the standalone or separatecomputer or display monitor; the OHMD can display non-surgical fieldrelated information and non-standalone or separate computer or displaymonitor related or displayed information when the user or surgeon isneither looking at the surgical field nor at the standalone or separatecomputer or display monitor or when the surgical field and/or thestandalone or separate computer or display monitor occupy only a portionof the visual field covered by the OHMD display. The switching ortoggling between surgical field related information, standalone orseparate computer or display monitor information and other informationby the OHMD display can be automatic, for example via image capture andrelated image processing and recognition which area the user or surgeonis currently looking at (optionally demarcated by optical markers,navigation markers including infrared markers, retroreflective markers,RF markers, and/or LEDs, or it can be via commands executed by the useror surgeon, e.g. voice commands or finger/keyboard commands, for exampleusing a virtual keyboard displayed by the OHMD display.

The OHMD can display information related to the information displayed onthe standalone or separate computer display or monitor in two dimensionsor three dimensions, the latter stereoscopically ornon-stereoscopically. Any number of combinations of displays can beapplied between the display by the OHMD display and the display by thestandalone or separate computer or monitor display. For example, whenthe computer or monitor displays shows a pre-operative orintra-operative imaging study of the patient, these can be displayed in2D (e.g. cross-sectional) or 3D using pseudo-3D display techniques, forexample with surface reconstruction and shading. Overlaying orsuperimposing, for example, a true 3D, e.g. stereoscopic 3D, view of theanatomy from the pre- or intra-operative imaging study and/or virtualsurgical plan of the patient using the OHMD display onto the sameanatomic structures and/or virtual surgical plan displayed in 2D orpseudo 3D by the standalone or separate computer or display monitor canbe beneficial for the surgeon as he or she executes surgical plans orplans next surgical plans during a procedure.

In some embodiments, the display of the OHMD unit or the standalone orseparate computer or display monitor can display functional and/or timestudies of the patient, e.g. the surgeon moving a leg or an arm of thepatient using physical-time fluoroscopic imaging, while the other of thetwo display modalities can simultaneously display and/or superimposestatic images. For example, the standalone or separate computer ordisplay monitor can display 2D or 3D function and/or time studies, e.g.of knee motion captured using physical-time 2D single or biplanefluoroscopy or captured using 3D CT fluoroscopy, while the display ofthe OHMD unit can superimpose 2D or 3D non-stereoscopic or 3Dstereoscopic images of the corresponding anatomy.

The following is an exemplary list of select possible combinations of2D, 3D non-stereoscopic and stereoscopic displays by the OHMD and 2D andpseudo 3D displays of the standalone or separate computer or displaymonitor. The list in Table 8 is in no way meant to be limiting of theinvention.

TABLE 8 Examples of possible combinations of display modes or types bythe display of the OHMD unit and the display of the standalone orseparate computer or display monitor. OHMD Display Standalone orSeparate 3D Non- 3D Computer or Display Monitor StereoscopicStereoscopic 2D with Pseudo 3D 3D Non- 3D with Function/ with Function/Pseudo Function/ with Function/ 2D Stereoscopic Stereoscopic Time Time2D 3D Time Time X X X X X X X X X X X X X X X X X X X X X X X X X X X XX X X X X X X X X X X X X denotes type of display mode used

The OHMD display can optionally display some virtual data, e.g.pre-operative images and/or image reconstructions, of the patient in 2D,while it can display other virtual data, e.g. aspects or components ofthe virtual plan, e.g. intended cut planes, in 3D. Similarly, the OHMDdisplay can optionally display some virtual data, e.g. pre-operativeimages and/or image reconstructions, of the patient in 3D, while it candisplay other virtual data, e.g. aspects or components of the virtualplan, e.g. intended pin or drill placement, in 2D, e.g. as a line.

The standalone or separate computer or display monitor can optionallydisplay some virtual data, e.g. pre-operative images and/or imagereconstructions, of the patient in 2D, while it can display othervirtual data, e.g. aspects or components of the virtual plan, e.g.intended cut planes, in pseudo 3D, e.g. with perspective views andshading. Similarly, the standalone or separate computer or displaymonitor can optionally display some virtual data, e.g. pre-operativeimages and/or image reconstructions, of the patient in 3D, while it candisplay other virtual data, e.g. aspects or components of the virtualplan, e.g. intended pin or drill placement, in 2D, e.g. as a line.

Aspects or components of the virtual surgical plan can, for example,include one or more of the following: a predetermined start point,predetermined start position, predetermined start orientation oralignment, predetermined intermediate point(s), predeterminedintermediate position(s), predetermined intermediate orientation oralignment, predetermined end point, predetermined end position,predetermined end orientation or alignment, predetermined path,predetermined plane, predetermined cut plane, predetermined contour oroutline or cross-section or surface features or shape or projection,predetermined depth marker or depth gauge, predetermined angle ororientation or rotation marker, predetermined axis, e.g. rotation axis,flexion axis, extension axis, predetermined axis of the virtual surgicaltool, virtual surgical instrument including virtual surgical guide orcut block, virtual trial implant, virtual implant component, implant ordevice, non-visualized portions for one or more devices or implants orimplant components or surgical instruments or surgical tools, and/or oneor more of a predetermined tissue change or alteration and/or one ormore of a predetermined position and/or orientation of the virtualsurgical tool, virtual surgical instrument including virtual surgicalguide or cut block, virtual trial implant, virtual implant component,implant or device.

In an additional embodiment, the OHMD display can optionally displaysome of the aspects or components of the virtual surgical plan in 2D andother aspects and components in 3D, stereoscopic or non-stereoscopic.For example, the OHMD display can display a intended cut plane in 3Dstereoscopic or non-stereoscopic, while it can display a virtual cutblock as an outline in 2D, for example projected with a stereoscopic 3Dview of the underlying tissue to be cut, e.g. a femoral neck for a hipreplacement. The OHMD display can display a virtual surgical instrument,e.g. a reamer in 3D, e.g. stereoscopic or non-stereoscopic, and it canproject the intended reaming axis in 2D or in 3D.

The standalone or separate computer or display monitor can optionallyco-display some of the aspects or components of the virtual surgicalplan in 2D and other aspects and components in pseudo 3D, optionallywith different colors. For example, the standalone or separate computeror display monitor can display a intended cut plane in pseudo 3D, whileit can display a virtual cut block as an outline in 2D, for exampleprojected on a pseudo 3D view of the underlying tissue to be cut, e.g. adistal femur for a knee replacement. The standalone or separate computeror display monitor can display a virtual implant or trial implant inpseudo 3D, and it can project its intended central axis, e.g. a femoralshaft axis for a femoral component of a hip replacement, in 2D.

The different 2D and 3D displays by the OHMD display and the standaloneor separate computer or display monitor can be displayed and viewedsimultaneously, in many embodiments substantially or partiallysuperimposed. Since the user or surgeon can view the standalone orseparate computer or display monitor through the OHMD display, the useror surgeon can experience a combination of 2D and 3D displayinformation, e.g. of virtual anatomy of the patient and/or aspects ofthe virtual surgical plan, not previously achievable.

TABLE 9 Further examples of possible combinations for simultaneousviewing of display modes or types by the display of the OHMD unit andthe display of the standalone or separate computer or display monitorfor virtual data of the patient including anatomy, e.g. pre- operativeimaging, and/or aspects and/or components of a virtual surgical plan,and/or virtual surgical instruments and/or virtual implants or implantcomponents and/or intra-operative imaging of the patient. Standalone orSeparate Computer or Display Monitor Virtual Anatomic Components ofVirtual Virtual Surgical Data of the Patient Surgical Plan of thePatient Instruments Pseudo Pseudo Pseudo 2D with 3D with 2D with 3D with2D with 3D with Pseudo Function/ Function/ Pseudo Function/ Function/Pseudo Function/ Function/ OHMD Display 2D 3D Time Time 2D 3D Time Time2D 3D Time Time Virtual Anatomic Data of the Patient 2D X X X X X X X XX X X X 3D Non-Stereoscopic X X X X X X X X X X X X 3D Stereoscopic X XX X X X X X X X X X 3D Non-Stereoscopic X X X X X X X X X X X X withFunction/Time 3D Stereoscopic with X X X X X X X X X X X X Function/TimeComponents of Virtual Surgical Plan of the Patient 2D X X X X X X X X XX X X 3D Non-Stereoscopic X X X X X X X X X X X X 3D Stereo scopic X X XX X X X X X X X X 3D Non-Stereoscopic X X X X X X X X X X X X withFunction/Time 3D Stereoscopic with X X X X X X X X X X X X Function/TimeVirtual Surgical Instruments 2D X X X X X X X X X X X X 3DNon-Stereoscopic X X X X X X X X X X X X 3D Stereoscopic X X X X X X X XX X X X 3D Non-Stereoscopic X X X X X X X X X X X X with Function/Time3D Stereoscopic with X X X X X X X X X X X X Function/ Time VirtualImplant or Trial Implant Components 2D X X X X X X X X X X X X 3DNon-Stereoscopic X X X X X X X X X X X X 3D Stereoscopic X X X X X X X XX X X X 3D Non-Stereoscopic X X X X X X X X X X X X with Function/Time3D Stereoscopic with X X X X X X X X X X X X Function/TimeIntra-Operative Imaging of the Patient 2D X X X X X X X X X X X X 3DNon-Stereoscopic X X X X X X X X X X X X 3D Stereoscopic X X X X X X X XX X X X 3D Non-Stereoscopic X X X X X X X X X X X X with Function/Time3D Stereoscopic with X X X X X X X X X X X X Function/Time Standalone orSeparate Computer or Display Monitor Virtual Implant or TrialIntra-Operative Imaging Implant Components of the Patient Pseudo Pseudo2D with 3D with 2D with 3D with Pseudo Function/ Function/ PseudoFunction/ Function/ OHMD Display 2D 3D Time Time 2D 3D Time Time VirtualAnatomic Data of the Patient 2D X X X X X X X X 3D Non-Stereoscopic X XX X X X X X 3D Stereoscopic X X X X X X X X 3D Non-Stereoscopic X X X XX X X X with Function/Time 3D Stereoscopic with X X X X X X X XFunction/Time Components of Virtual Surgical Plan of the Patient 2D X XX X X X X X 3D Non-Stereoscopic X X X X X X X X 3D Stereo scopic X X X XX X X X 3D Non-Stereoscopic X X X X X X X X with Function/Time 3DStereoscopic with X X X X X X X X Function/Time Virtual SurgicalInstruments 2D X X X X X X X X 3D Non-Stereoscopic X X X X X X X X 3DStereoscopic X X X X X X X X 3D Non-Stereoscopic X X X X X X X X withFunction/Time 3D Stereoscopic with X X X X X X X X Function/ TimeVirtual Implant or Trial Implant Components 2D X X X X X X X X 3DNon-Stereoscopic X X X X X X X X 3D Stereoscopic X X X X X X X X 3DNon-Stereoscopic X X X X X X X X with Function/Time 3D Stereoscopic withX X X X X X X X Function/Time Intra-Operative Imaging of the Patient 2DX X X X X X X X 3D Non-Stereoscopic X X X X X X X X 3D Stereoscopic X XX X X X X X 3D Non-Stereoscopic X X X X X X X X with Function/Time 3DStereoscopic with X X X X X X X X Function/Time X denotes type ofdisplay mode combinations used or possible

Virtual data of the patient including anatomy, e.g. pre-operativeimaging, and/or aspects and/or components of a virtual surgical plan,and/or virtual surgical instruments and/or virtual implants or implantcomponents and/or intra-operative imaging of the patient can bedisplayed using different colors, greyscale values and image intensitiesby the display of the OHMD unit and the display of the standalone orseparate computer or display monitor.

Intra-operative imaging of the patient can include, for example, x-rayimaging, laser scanning, 3D scanning or mechanical probe scanning of ajoint, e.g. hip joint, knee joint, shoulder joint, or a spine.Intra-operative X-ray images, laser scans, 3D scans, mechanical probescans, pre-operative imaging data of the patient including 2D and 3Dreconstructions, aspects or components of a virtual surgical plan,virtual surgical instruments, and/or virtual implants and implantcomponents can be displayed simultaneously and, optionally, superimposedby the display of the OHMD unit and the display of the standalone orseparate computer or display monitor. If two or more imaging modalitiesor pre-operative and intra-operative imaging studies are co-displayed,they can optionally be anatomically matched and they can optionally bedisplayed using the same projection plane or, optionally, differentprojection planes.

If 2D views are co-displayed with 3D views or pseudo 3D views by theOHMD display alone, by the standalone or separate computer or displaymonitor alone, or the two together and partially or completelysuperimposed, the 2D views can optionally be displayed using certainstandard projections, e.g. AP, lateral, oblique; the standardprojection, e.g. AP, lateral and oblique, can optionally be referencedto the live data of the patient, e.g. the corresponding planes with thepatient positioned on the OR table,

-   -   or to the data of the patient displayed on the standalone or        separate computer or display monitor. Standard projections or        standard views can also include view angles from the patient's        side, front, top, bottom, or oblique views.

Dynamic views or functional views, for example with two or three spatialdimensions and a time dimension can be displayed by the display of theOHMD unit and/or the display of the standalone or separate computer ordisplay monitor, optionally superimposed onto or co-displayed withstatic images, e.g. 2D or 3D, by the second display unit, e.g. thedisplay of the OHMD unit or the display of the standalone or separatecomputer or display monitor. Such dynamic views or functional views caninclude kinematic studies of a joint, e.g. obtained with anintraoperative laser or 3D scanner, which can be used by a surgeon toobtain scans of the knee, hip, shoulder an any other joint at differentflexion angles, extensions angles, rotation angles, abduction angles,adduction angles, e.g. 0, 10, 15, 20, 30, 40, 45, 50, 60, 70, 80, 90,100, 110, 120, 130, 140, 150 degrees etc. Any other type of dynamicscan, which can include a time element or time dimension or a functionalelement or functional dimension can be displayed by the display of theOHMD unit and/or the display of the standalone or separate computer ordisplay monitor.

In some embodiments of the invention, the display of the OHMD unit canbe used for displaying lower resolution data and/or images, while thedisplay of the display of the standalone or separate computer or displaymonitor can be used for displaying corresponding or matching oroverlapping higher resolution data and/or images. This embodiment can beparticularly useful when, for example, the maximum available displayresolution of the OHMD is lower than desirable for a particularapplication or surgical procedure. This embodiment can also be useful,when the software environment limits, for example, the amount of surfacepoints or nodes displayed or limits the available resolution. Thisembodiment can also be useful when a WiFi or Bluetooth or other wirelessconnection is used with the OHMD with limitations in bandwidth and/ordata transmission, thereby limiting the amount of data being transmittedto the OHMD and, ultimately, displayed, in particular when thislimitation implies a limitation in available spatial resolution for thedisplay of the data and/or images by the OHMD. By viewing the lowerresolution data and/or images through the OHMD, the user can have, forexample, the benefit of stereoscopic visualization or the benefit ofviewing components or aspects of the surgical plan, e.g. a virtualresection line, a virtual cut plane, a virtual instrument and/or avirtual implant, while by viewing simultaneously and/or with partial orcomplete superimposition the higher resolution data and/or images on thedisplay of the standalone or separate computer or display monitor theviewer can have the concurrent benefit of viewing the data and/or imagesin high resolution.

In some embodiments of the invention, the display of the OHMD unit canbe used for displaying static data and/or images, while the display ofthe standalone or separate computer or display monitor can be used fordisplaying corresponding or matching or overlapping dynamic data and/orimages, e.g. images demonstrating a function, e.g. kinematic movement ofa joint, and/or a time element or dimension including a change incondition or function monitored over a time period. This embodiment canbe particularly useful when, for example, the refresh rate of the OHMDdisplay is lower than desirable for a particular application or surgicalprocedure. This embodiment can also be useful, when the softwareenvironment limits, for example, the amount of data and/or imagesdisplayed. This embodiment can also be useful when a WiFi or Bluetoothor other wireless connection is used for connecting the OHMD withlimitations in bandwidth and/or data transmission, thereby limiting theamount of data being transmitted to the OHMD and, ultimately, displayed,in particular when this limitation implies a limitation in availabletemporal and/or spatial resolution for the display of the data and/orimages by the OHMD. By viewing the static data and/or images through theOHMD, the user can have, for example, the benefit of stereoscopicvisualization or the benefit of viewing components or aspects of thesurgical plan, e.g. a virtual resection line, a virtual cut plane, avirtual instrument and/or a virtual implant, while by viewingsimultaneously and/or with partial or complete superimposition thedynamic data and/or images on the display of the standalone or separatecomputer or display monitor the viewer can have the concurrent benefitof viewing the dynamic data and/or images, optionally in highresolution.

In some embodiments of the invention, the display of the OHMD unit canbe used for displaying a subset of the data and/or images representing asmaller portion of the field of view displayed by the standalone orseparate computer or display monitor, while the display of the displayof the standalone or separate computer or display monitor can be usedfor displaying corresponding or matching or overlapping higher dataand/or images using the full intended field of view of patient data.This embodiment can, for example, be useful, when the softwareenvironment limits the amount of surface points or nodes displayed orlimits the size of the data displayed by the OHMD. This embodiment canalso be useful when a WiFi or Bluetooth or other wireless connection isused with the OHMD with limitations in bandwidth and/or datatransmission, thereby limiting the amount of data being transmitted tothe OHMD and, ultimately, displayed, in particular when this limitationimplies a limitation in the amount of data available for the display ofthe data and/or images by the OHMD. By viewing data and/or images with asmaller, more narrow field of view through the OHMD, the user can have,for example, the benefit of stereoscopic visualization or the benefit ofviewing components or aspects of the surgical plan, e.g. a virtualresection line, a virtual cut plane, a virtual instrument and/or avirtual implant, while by viewing simultaneously and/or with partial orcomplete superimposition the data and/or images with the full field ofview on the display of the standalone or separate computer or displaymonitor the viewer can have the concurrent benefit of viewing the dataand/or images using the full intended field of view of patient data.When 3D views are superimposed onto or co-displayed with 2D views by thedisplay of the OHMD unit and the display of the standalone or separatecomputer or display monitor or when multiple 2D views are superimposedor co-displayed by the display of the OHMD unit and the display of thestandalone or separate computer or display monitor, they can beanatomically matched, for example using corresponding landmarks and/orusing common coordinates. They can also have different view angles, e.g.a view angle as the patient is positioned on the OR table, a view anglefrom the side, front, top, bottom, or oblique views. Thus, the OHMDdisplay can, for example, show a stereoscopic 3D view of the patient'svirtual anatomy, e.g. from a pre-operative imaging study, while thestandalone or separate computer or display monitor can show a matchingAP or lateral intra-operative radiographic view or a matching pseudo 3Dlaser view of the patient.

The matching of data displayed by the display of the OHMD unit and thedisplay of the standalone or separate computer or display monitor can beachieved in different ways, e.g. using

-   -   Matching of data and/or image using coordinates    -   Matching of data and/or image using content    -   Combinations of matching of data and/or image coordinates and        data and/or image content

In some embodiments of the invention, data and/or images displayed bythe OHMD and data and/or images displayed by the standalone or separatecomputer or display monitor can be matched using known image coordinatesand can then optionally be partially or completely superimposed, e.g. asthe user and/or surgeon moves his or her head and/or body while lookingat the standalone or separate computer or display monitor. For example,if the OHMD is registered in space, e.g. with regard to the patientand/or the surgical site and/or the standalone computer or displaymonitor, data and/or images displayed by the OHMD and/or displayed bythe standalone computer or display monitor can be in the same or acommon coordinate system, which can allow the matching orsuperimposition of the display by the OHMD with the display by thestandalone or separate computer or display monitor, when portions or allof the separate computer or display monitor are included in the field ofview of the user or surgeon through the OHMD.

In some embodiments of the invention, when both the display of the OHMDand the display of the separate computer or display monitor areregistered in the same coordinate system, the OHMD can display then aset of data and/or images at least partially matching the coordinates ofthe data and/or images of the separate computer or display monitor. Forexample, the OHMD can display stereoscopic 3D views that share commoncoordinates with a pseudo 3D visualization displayed by the standaloneor separate computer or display monitor. Such common coordinates can,for example, be corner points or edges or select geometric featuresand/or locations which can be superimposed then in the resultantcomposite OHMD/standalone monitor view that the user or surgeon sees.The OHMD can also, for example, display a stereoscopic 3D view of livedata of the patient or virtual data of the patient or both, while thestandalone or separate computer or display monitor displays a 2D view,e.g. a pre-operative imaging study, of the patient. The 2D plane or viewdisplay by the standalone or separate computer or display monitor canhave the same or common coordinates with the corresponding 2D planeembedded in or contained in the 3D data and/or images displayed by theOHMD which can be matched or superimposed then in the resultantcomposite OHMD/standalone monitor view that the user or surgeon sees.Alternatively, in a similar example, if the OHMD provides only a surfacedisplay, for example, the periphery or outline or select peripheralpoints of the 2D plane displayed by the standalone or separate computeror display monitor can have the same or common coordinates withcorresponding surface points in the location corresponding to the 2Dplane in the 3D data and/or images displayed by the OHMD.

The data and/or images displayed by the OHMD can be matched to the datadisplayed by the standalone or separate computer or display monitor,e.g. by identifying common coordinates and superimposing them and/or bydefining a common coordinate system. Alternatively, the data and/orimages displayed by the standalone or separate computer or displaymonitor can be matched to the data displayed by the OHMD, e.g. byidentifying common coordinates and superimposing them and/or by defininga common coordinate system. When the data and/or images displayed by theOHMD are superimposed with the data and/or images displayed by thestandalone or separate display monitor, the data and/or images displayedby the OHMD and the data and/or images displayed by the standalone orseparate display monitor can be displayed with the same magnification inorder to optimize the superimposition or matching.

In some embodiments of the invention, the surgical table can be moved.The movement of the surgical table can translate into a comparablemovement of the patient and/or the surgical site in x, y, and/or zdirection. When the magnitude and direction of the table movement isknown, it can be used to move the common coordinate system by acorresponding amount or direction for matching or superimposing the dataand/or images displayed by the OHMD and the data and/or images displayedby the standalone or separate display monitor. For example, if the OHMDdisplays live data of the patient, e.g. captured through an image and/orvideo capture system integrated into, attached to or separate from theOHMD, and/or virtual data of the patient and/or virtual data of thepatient superimposed onto live data of the patient and the standalone orseparate computer or display monitor displays a pre-operative imagingstudy of the patient, the surgical table and the patient can be movedand the display of the live or virtual data by the OHMD can be moved bya corresponding amount, thereby maintaining registration includingregistration to the data displayed on the standalone or separatecomputer or display monitor.

In some embodiments of the invention, the data and/or images displayedby the OHMD and the data and/or images displayed by the standalone orseparate computer or display monitor can be cross-registered and, forexample, moved into a shared or common coordinate system with use of animage and/or video capture system integrated into, attached to, orseparate from the OHMD, capturing the data displayed by the standaloneor separate computer or display monitor. For example, the standalone orseparate computer or display monitor can display data from aphysical-time intra-operative imaging study of the patient, including,for example, imaging during movement of the patient or surgical table orboth. Standard image processing techniques can, for example, recognizeanatomic landmarks or features on the data or images displayed on thestandalone or separate computer or display monitor and match these withthe corresponding anatomic landmarks or features in the data and/orimages available for display by the OHMD. The OHMD can then display thecorresponding data and/or images, optionally superimposing the databased on landmark matching. The landmark matching can, for example,occur by moving and/or translating the data or images available fordisplay by the OHMD by an amount that will superimpose or match in acommon coordinate system corresponding anatomic landmarks and/orfeatures.

In some embodiments, the same process can be applied without using theimage and/or video capture system by directly comparing images acquiredwith the physical-time intra-operative imaging system with data and/orimages available for display by the OHMD. Standard image processingtechniques can, for example, recognize anatomic landmarks or features onthe data or images acquired by the physical-time imaging system andmatch these with the corresponding anatomic landmarks or features in thedata and/or images available for display by the OHMD. The OHMD can thendisplay the corresponding data and/or images, optionally superimposingthe data based on landmark matching. The landmark matching can, forexample, occur by moving and/or translating the data or images availablefor display by the OHMD by an amount that will superimpose or match in acommon coordinate system corresponding anatomic landmarks and/orfeatures.

In the foregoing embodiments, the data and/or images displayed by theOHMD can be matched to the data displayed by the standalone or separatecomputer or display monitor, e.g. by identifying common coordinates andsuperimposing them and/or by defining a common coordinate system.Alternatively, the data and/or images displayed by the standalone orseparate computer or display monitor can be matched to the datadisplayed by the OHMD, e.g. by identifying common coordinates andsuperimposing them and/or by defining a common coordinate system. Whenthe data and/or images displayed by the OHMD are superimposed with thedata and/or images displayed by the standalone or separate displaymonitor, the data and/or images displayed by the OHMD and the dataand/or images displayed by the standalone or separate display monitorcan be displayed with the same magnification in order to optimize thesuperimposition or matching.

Matching of images displayed by the OHMD and a standalone or separatecomputer or display monitor can also be performed by combiningcoordinate based matching, e.g. using the same coordinate system forboth displays, and landmark based matching using any of the foregoingtechniques. Someone skilled in the art will readily recognize othermeans of coordinate matching and landmark matching.

In some embodiments of the invention, the magnification of the itemsdisplayed by the OHMD can be adjusted so that it is reflective of,corresponds to, is smaller or larger than the magnification used by thestandalone or separate computer or display monitor. Alternatively, thestandalone or separate computer or display monitor can have one or moremarkers, e.g. one or more LEDs, that an image and/or video capturesystem, e.g. integrated into, attached to or separate from the OHMD, candetect which, in turn, can then trigger the adjustment of themagnification of the items displayed by the OHMD. In some embodiments,an image and/or video capture system integrated into, attached to orseparate from the OHMD can visualize the size and shape (round, oval,ellipsoid, rectangular, square) of the standalone or separate computeror display monitor; using standard image processing techniques andgeometry, the size and shape can then be used to derive the distance andangle of the OHMD relative to the standalone or separate computer ordisplay monitor. If more than one camera is used, additional parallaxinformation (difference in size and/or shape of the standalone orseparate computer or display monitor) can be used to further estimate orimprove the estimation of the distance or angle of the OHMD to thestandalone or separate computer or display monitor. The resultantestimation of the distance and/or angle of the OHMD display to thestandalone or separate computer or display monitor can then optionallybe used to match the magnification of the data displayed by thestandalone or separate computer or display monitor or to display at ahigher or lower magnification than the data display by the standalone orseparate computer or display monitor.

Similarly, the OHMD can detect, e.g. automatically, if the surgeon oroperator is not looking at the standalone or separate computer ordisplay monitor, for example, with use of an image and/or video capturesystem integrated into, attached to or separate from the OHMD. The imageand/or video capture system can, for example, detect that the outline ofthe standalone or separate computer or display monitor (e.g. round,square, rectangular) is not present in the captured image data and thesoftware can the automatically adjust the magnification of the itemsdisplayed by the OHMD so that it is reflective of or corresponds to thedistance of the OHMD or the surgeon's eyes to the patient's surgicalsite, or is smaller or larger than that. Alternatively, a standalone orseparate computer or display monitor can have one or more markers, e.g.one or more LEDs or optical markers, that the image and/or video capturesystem can detect; in this case, when the image captures system noticesthat the one or more LEDs or optical markers are not included in theimage capture data, the software can then automatically adjust themagnification of the items displayed by the OHMD so that it isreflective of or corresponds to the distance of the OHMD or thesurgeon's eyes to the patient's surgical site, or is smaller or largerthan that. Similarly, markers or LEDs placed on the patient's surgicalsite can be detected by the OHMD including an image and/or video capturesystem integrated into, attached to or separate from the OHMD therebytriggering an adjustment in magnification so that it is reflective of,corresponds to the distance of the OHMD or the surgeon's eyes to thepatient's surgical site, or is smaller or larger than that when thesurgeon or operator is looking at the patient's surgical site.

In some embodiments of the invention, the OHMD can be used to displaydata and/or images instead of a standalone or separate computer ordisplay monitor. Optionally, the OHMD can replace the standalone orseparate computer or display monitor. In some embodiments, the OHMD candisplay the live data from the patient's surgical site and project themfor the surgeon and superimpose them with virtual data. The OHMD canalso display one or more aspects or components of the virtual surgicalplan, e.g. projected paths for one or more surgical instruments, or itcan display one or more virtual implants or implant components. In thisembodiment, the OHMD can optionally match the magnification of the oneor more projected paths, and/or one or more surgical instruments and/orone or more virtual implants or implant components relative to themagnification of the live data from the patient. The OHMD can also applya larger or smaller magnification and/or size than the magnification ofthe live data from the patient for the one or more projected pathsand/or virtual surgical instruments, and/or one or more virtual implantsor implant components. The live data of the patient can be seen throughthe transparent display of the OHMD. Alternatively, the display can bepartially or completely opaque and the live data can be capture throughan image and/or video capture system integrated into, attached to orseparate from the OHMD and then subsequently be displayed by the OHMDdisplay.

In some embodiments of the invention, for example when the OHMD is theprimary display unit, the OHMD can be non-transparent to light orminimally transparent to light reflected from the patient's surgicalfield and can display, for example, live (electronic) images collectedby the image and/or video capture system and, optionally, it candisplay, in addition, aspects or components of the virtual surgicalplan, e.g. one or more projected paths for one or more physical surgicalinstruments, probes, pointers, and/or one or more virtual instrumentsand/or one or more virtual implants or implant components (optionallywith various chosen matching or non-matching magnifications). In thissetting, the OHMD can also display electronic images of the physicalsurgical instruments and or devices and their respective movements, forexample captured with an image and/or video capture system integratedinto, attached to, or separate from the OHMD (with various chosenmatching or non-matching magnifications).

The OHMD can be permanently non-transparent to light or minimallytransparent to light reflected from the patient's surgical field.Alternatively, the degree of transparency can be variable, for examplewith use of one or more optical filters, e.g. polarizing light filters,in front of or integrated into the OHMD or electronic, e.g. LCD, oroptical filters in front or integrated into the OHMD, or via intensityadjustments. The OR theater can optionally use light sources, e.g.polarized or filtered light that will support modulation or aid withadjustments of the transparency of the OHMD to light reflected from thepatient's surgical field.

Magnified Displays

Magnified displays of the following structures and/or devices can beshown with an OHMD for example for one or more of the following,simultaneously or non-simultaneously:

-   -   Physical anatomy (e.g. using intra-operative imaging with        optional magnification or demagnification)        -   Static        -   Dynamic, e.g. with functional or time element or dimension    -   Virtual anatomy, e.g. from pre-operative imaging study    -   Aspects or components of a virtual surgical plan, e.g. a        predetermined start point, predetermined start position,        predetermined start orientation or alignment, predetermined        intermediate point(s), predetermined intermediate position(s),        predetermined intermediate orientation or alignment,        predetermined end point, predetermined end position,        predetermined end orientation or alignment, predetermined path,        predetermined plane, predetermined cut plane, predetermined        contour or outline or cross-section or surface features or shape        or projection, predetermined depth marker or depth gauge,        predetermined angle or orientation or rotation marker,        predetermined axis, e.g. rotation axis, flexion axis, extension        axis, predetermined axis of the virtual surgical tool, virtual        surgical instrument including virtual surgical guide or cut        block, virtual trial implant, virtual implant component, implant        or device, non-visualized portions for one or more devices or        implants or implant components or surgical instruments or        surgical tools, and/or one or more of a predetermined tissue        change or alteration and/or one or more of a predetermined        position and/or orientation of the virtual surgical tool,        virtual surgical instrument including virtual surgical guide or        cut block, virtual trial implant, virtual implant component,        implant or device    -   Virtual surgical instrument(s)    -   Virtual implant(s) or implant component(s)

In some embodiments of the invention, the OHMD display can display livedata of the patient captured through an image and/or video capturesystem integrated into, attached to or separate from the OHMD withhigher magnification than the live data seen through transparentportions of the OHMD by the user's or surgeon's eye. Thus, the live dataof the patient captured through an image and/or video capture systemintegrated into, attached to or separate from the OHMD can be displayedin a magnified manner for a given distance of the OHMD display to thesurgical field. This has the benefit that select structures can be seenwith greater detail, for example offering a low power microscopic,magnified view of portions or all of the surgical field. The distance ofthe OHMD to the surgical field can be determined using techniquesdescribed in the specification, e.g. optical markers, navigation markersincluding infrared markers, retroreflective markers, RF markers, IMUs,LEDs and any other technique known in the art.

The magnified display of live data can be performed while partially orcompletely blending out live data seen through the OHMD, e.g. with theOHMD turned partially or completely opaque to light emitted from thesurgical field and primarily or only data displayed captured through theimage and/or video capture system. The magnified display of live datacaptured through the image and/or video capture system can besuperimposed on live data seen through one or more partially orcompletely transparent portions of the OHMD. In this example, themagnified display of the live data can be a portion of the surgicalfield seen through the OHMD.

Optionally, a declining gradient of magnification can be applied to thelive data so that the magnified live data can blend in seamlessly ornear seamlessly with the non-magnified live data, e.g. the live dataseen through one or more partially or completely transparent portions ofthe OHMD.

The magnification of a portion or all of the live data captured throughan image and/or video capture system can be at preset levels, e.g. 1.5×,2.0×, 3.0×, 4.0×, or 5.0× or any other magnification level. Themagnification can be continuous, e.g. on a sliding scale. Themagnification can be selected by the user and/or surgeon, for exampleusing voice commands, eye commands or using a virtual keyboard interfacedisplayed by the OHMD.

Virtual data can optionally be displayed with the same magnification asthe live data. Optionally, virtual data can be displayed with nomagnification or lesser or greater magnification than live data.

In some embodiments of the invention, the OHMD display can displayvirtual data of the patient with higher magnification than the live dataseen through transparent portions of the OHMD by the user's or surgeon'seye. Thus, the virtual data of the patient can be displayed in amagnified manner for a given distance of the OHMD display to thesurgical field. This has the benefit that select structures or aspectsof components of a virtual surgical plan can be seen with greaterdetail, for example offering a low power microscopic, magnified view ofportions or all of the virtual data. The distance of the OHMD to thesurgical field can be determined using techniques described in thespecification, e.g. optical markers, navigation markers includinginfrared markers, retroreflective markers, RF markers, IMUs, LEDs andany other technique known in the art.

The magnified display of virtual data can be performed while partiallyor completely blending out live data seen through the OHMD, e.g. withthe OHMD turned partially or completely opaque to light emitted from thesurgical field and primarily or only virtual data displayed. Themagnified display of virtual data captured through the image and/orvideo capture system can be superimposed on live data seen through oneor more partially or completely transparent portions of the OHMD. Inthis example, the magnified display of the virtual data can be a portionof the surgical field seen through the OHMD.

Optionally, a declining gradient of magnification can be applied to thevirtual data so that the magnified virtual data can blend in seamlesslyor near seamlessly with the non-magnified live data, e.g. the live dataseen through one or more partially or completely transparent portions ofthe OHMD.

The magnification of a portion or all of the virtual data can be atpreset levels, e.g. 1.5×, 2.0×, 3.0×, 4.0×, or 5.0× or any othermagnification level. The magnification can be continuous, e.g. on asliding scale. The magnification can be selected by the user and/orsurgeon, for example using voice commands, eye commands or using avirtual keyboard interface displayed by the OHMD.

Both portions or all of live data and virtual data can be displayedusing magnification or no magnification. Non-limiting examples ofpossible magnification combinations between live data and virtual dataare provided below.

TABLE 10 Exemplary, non-limiting combinations of magnifications of livedata and/or virtual data. Live data, e.g. as captured by image capturesystem and displayed by OHMD Virtual data Original size Portionsmagnified All magnified Portions minified All minified Original size X XX X X Portions magnified X X X X X All magnified X X X X X Portionsminified X X X X X All minified X X X X X X denotes type ofmagnification mode combinations used or possible

The magnification of live data and virtual data can be the same. Themagnification of live data and virtual data can be different. Virtualdata can be partially, e.g. affecting only part of the displayed virtualdata, or all magnified. Live data can be partially, e.g. affecting onlypart of the displayed live data, or all magnified. Virtual data can bemagnified while live data are not magnified. Live data can be magnifiedwhile virtual data are not magnified. Any combination is possible.

The term magnification includes also displays wherein the live data orthe virtual data are displayed in a format or with a magnification thatis smaller than live data seen through transparent portions of the OHMDfor a given distance.

The magnification can be applied around a central point, e.g. an anchorpoint, an anatomic landmark, a pin entry into a bone, a screw head, orcentral axis of the field of view of the OHMD, a pin axis or a screwaxis. When a central point is used, the coordinates of the central pointin the live data of the patient as seen by the surgeon's right eyethrough the OHMD unit will be the same as the view coordinates of thecentral point in the virtual data of the patient seen by the surgeon'sright eye projected by the display of the OHMD unit; the coordinates ofthe central point in the live data of the patient as seen by thesurgeon's left eye through the OHMD unit will be the same as the viewcoordinates of the central point in the virtual data of the patient seenby the surgeon's left eye projected by the display of the OHMD unit.When a central axis is used, the coordinates of the central axis in thelive data of the patient as seen by the surgeon's right eye through theOHMD unit will be the same as the view coordinates of the central axisin the virtual data of the patient seen by the surgeon's right eyeprojected by the display of the OHMD unit; the coordinates of thecentral axis in the live data of the patient as seen by the surgeon'sleft eye through the OHMD unit will be the same as the view coordinatesof the central axis in the virtual data of the patient seen by thesurgeon's left eye projected by the display of the OHMD unit. Whenstereoscopic projection is used with the left and right displays of theOHMD unit, the view coordinates for the left display and the rightdisplay of the OHMD unit will be different for the left eye and theright eye; the difference in view coordinates is a reflection of theparallax. For example, when the user or surgeon elects to turn onmagnification of live and/or virtual data, the magnification can beapplied around the central point of the last unmagnified field of view.The system including its software can optionally apply the magnificationautomatically around the central point of the last field of view.Alternatively, the user and/or surgeon can use a different central pointor central axis as the center around which the live and/or virtual dataare being magnified. The central point or central axis can, for example,coincide with the center of a pedicle, when spinal surgery iscontemplated. The central axis can coincide with an acetabular orfemoral axis, e.g. an anteversion axis. The central axis can, forexample, be a predetermined path. The central point, can, for example,be an endpoint. The central point or central axis can, for example, bethe center of an acetabulum when hip replacement or other hip surgery iscontemplated. The central point or central axis can, for example, be thecenter of a glenoid when shoulder surgery is contemplated. The centralpoint or central axis for magnification can be pre-selected for variousanatomic sites or surgical fields or surgeries contemplated, e.g. hipreplacement, knee replacement surgery, knee arthroscopy or spinalfusion. Using, for example, one or more image and/or video capturesystems integrated into, attached to or separate from the OHMD, or usingintra-operative imaging, one or more anatomic structures can optionallybe identified using standard image processing techniques (e.g. theacetabulum and its center) and the central point or central axis for anymagnified views can optionally be set or defined automatically.

View Patient/View Computer Monitor/Screen

In some embodiments, the magnification of the OHMD display can bematched with the magnification of a computer monitor, e.g. in the OR, sothat corresponding tissues shown by the OHMD and/or the computer monitorare displayed using the same magnification and can, for example, besubstantially aligned or superimposed between the OHMD and the computermonitor display.

Displaying Surgical Instruments and/or Medical Devices/Implantables

In some embodiments of the invention, surgical instruments or medicaldevices or implantables can be displayed virtually with the live data ofthe patient. The virtual data surgical instrument or virtual implantablecan be shown by the OHMD superimposed onto the live data of the patientincluding the live data surgical instrument.

The OHMD can show the virtual surgical instrument or the virtualimplantable indicating the desired orientation or direction or placementof the virtual surgical instrument or the virtual implantable, forexample using a virtual surgical plan. Optionally, the OHMD can displaydirectional markers such as an intended path derived from a surgicalplan to help guide the surgeon direct the physical surgical instrumentor the physical implantable.

The physical surgical instrument or physical implantable can be scannedpreoperatively to derive its shape and/or dimensions for subsequentdisplay of a derived shape or dimension of a virtual representation ofthe surgical instrument or the implantable by the OHMD. Alternatively, aCAD file or 3D file of the surgical instrument or the implantable can beused. Preoperative scanning of the surgical instrument or theimplantable can be performed using any technique known in the art.Scanning of the surgical instrument or the implantable can be performedby the OHMD, for example using a built in image capture device. Scanningof the surgical instrument or the implantable can be performed by aseparate image capture device.

In some embodiments, scanning of the surgical instrument or theimplantable can occur in two or more dimensions. The more dimensions areused typically the more accurate the resultant virtual representation ofthe surgical instrument or the implantable.

If an image capture device is used, e.g. one attached to or integratedinto the OHMD or coupled to or separate from the OHMD, the surgicalinstrument or the implantable can be scanned in one, two or moreprojections, positions or orientation, e.g. by moving the OHMD or thesurgical instrument or implantable into different positions ororientations. In some embodiments, the surgical instrument or theimplantable can be placed on a tray or fixture for this purpose, whichallows to move the surgical instrument or the implantable into differentpositions and, optionally, to rotate the surgical instrument or theimplantable. In some embodiments, the distance between the surgicalinstrument or the implantable and the image capture device, including animage capture device attached to or integrated into the OHMD or coupledto or separate from the OHMD, is fixed, while the surgical instrument orthe implantable are being scanned.

Scans of the physical surgical instrument or implantable can then beused to derive a virtual 2D or 3D representation of the surgicalinstrument or the implantable.

By scanning the surgical instrument or the implantable intraoperatively,the surgeon has great flexibility in using different surgicalinstruments or implantables which he can change and modify and,optionally, integrate into his physical or virtual surgical plan.

The surgeon can optionally store each surgical instrument or implantablethat has been scanned in this manner in a virtual library of surgicalinstruments or implantables. The virtual surgical instruments orimplantables stored in this manner can be named and stored for futureuse in subsequent surgical procedures in other patients. By storing thevirtual surgical instruments or implantables the need for repeat scansof the same surgical instrument or same type or shape of implantable isobviated.

In some embodiments of the invention, the surgeon can use the virtualdata of the surgical instrument or implantables that were previouslygenerated in a new surgical plan for another, new patient. The surgeoncan select a desired virtual surgical instrument or implantable from thevirtual library and use the virtual surgical instrument or the virtualimplantable in his or her virtual surgical plan.

When the surgeon performs the physical surgery and the OHMD displaysoptionally the virtual surgical instrument or implantable, optionallysuperimposed onto or displayed near the physical surgical instrument orimplantable, the software can optionally compare the size and shape ofthe physical surgical instrument or implantable with that of thepreviously selected virtual surgical instrument or implantable.Alternatively, the surgeon can visually compare the size and/or shape ofthe virtual and the physical surgical instrument or implantable.

If a size and/or shape mismatch is detected, the software can send analert or alarm to the surgeon, e.g. visual or audible, that indicates amismatch. A mismatch can indicate to the surgeon that the accuracy ofregistration of virtual data and live data has been compromised and thatre-registration may be required. A mismatch can also indicate to thesurgeon that the wrong physical surgical instrument or implantable hasbeen selected in comparison to the previously identified virtualsurgical instrument or implantable. In this case, the surgeon can checkthe virtual surgical plan or the physical surgical plan and modifyeither or both, for example by selecting a different size or shapevirtual or live surgical instrument or implantable.

Stereoscopic and Non-Stereoscopic 3D Display of Virtual Data of thePatient with Superimposition on Live Data of the Patient

In some embodiments of the invention, the OHMD can display a virtual 2Dor 3D image of the patient's normal or diseased tissue or an organ or asurgical site or target tissue with a view angle or a perspective orprojection that is different for the display for the left eye comparedto the display for the right eye resulting in a stereoscopic projectionof the anatomy or the pathologic tissue. The virtual data of the patientis thus superimposed on the live data of the patient, e.g. the surgicalsite, for the left and right eye of the surgeon, respectively, usingboth the left and the right view angle for the surgeon. This means thattwo separate views are rendered from the virtual 2D or 3D data sets, onefor the left eye and one for the right eye. Multidimensional viewsexceeding three dimensions generated for the left eye and the right eyeare possible. For example, in addition to the virtual anatomy of thepatient vascular flow or joint motion can be displayed separately forthe left eye and the right eye. The difference in perspective betweenthe left eye and the right eye projection of virtual data or parallaxcan be selected or programmed so that it will change, for example, withthe distance of the OHMD, the surgeon's head or the surgeon's eye inrelationship to the target site, surgical site or target tissue. Thedistance between the surgeon's or operator's eyes can also be taken intoaccount. In some embodiments, the difference in perspective or parallaxwill be selected or programmed so that a 3D effect is generated in astereoscopic 3D manner or effect. The difference in perspective orparallax can change depending on any changes in the distance of theOHMD, the surgeon's or operator's head or the surgeon's or operator'seye in relationship to the target site, surgical site or target tissue.For example, as the surgeon or operator moves away from the target site,surgical site or target tissue, the difference in perspective orparallax can decrease. As the surgeon or operator moves towards thetarget site, surgical site or target tissue, the difference inperspective or parallax can increase. The decrease or increase can belinear, non-linear, exponential or algorithmic. Any other mathematicalfunction is possible. In some embodiments, the difference in perspectiveor parallax will change similar to the change experienced by the humaneye as the surgeon or operator moves towards or away from a target.

The distance of the OHMD, the surgeon's or operator's head or thesurgeon's or operator's eye in relationship to the target site, surgicalsite or target tissue can be measured via image capture, anatomiclandmark embodiments, image capture used in conjunction with calibrationor registration phantoms, surgical navigation or any of the otherembodiments described in this specification and or spatial mapping. Thedistance and any changes in distance of the OHMD, the surgeon's oroperator's head or the surgeon's or operator's eye in relationship tothe target site, surgical site or target tissue can be used to changethe difference in perspective views or parallax in views for the lefteye and the right eye.

FIGS. 16A and 16B are flow charts summarizing model generation,registration and view projection for one or more OHMDs, e.g. by aprimary surgeon, second surgeon, surgical assistant nurse, or others.Pre-operative, intra-operative or post-operative images of the patientcan be acquired 240. The image data can optionally be segmented 241. 3Dreconstructions of the patient's anatomy or pathology including multipledifferent tissues, e.g. using different colors or shading, can begenerated 242. Virtual 3D models of surgical instruments and devicescomponents can be generated which can include their predeterminedposition, location, rotation, orientation, alignment and/or direction243. The virtual 3D models can be registered, for example inrelationship to the OHMD and the patient 244. The virtual 3D models canbe registered relative to the live patient data 245. Optionally,adjustments can be made for different view perspectives, parallax, skin,skin movement and other tissue specific issues 246. Differentperspective views can be generated for the user's left eye and right eyeto facilitate a stereoscopic viewing experience, e.g. like an electronichologram, of the virtual models of subsurface or hidden anatomic orpathologic tissues 247 and the virtual 3D models of tools, instruments,implants and devices 248. Virtual patient data 249 and virtual 3D modelsof tools, instruments, implants and devices 250 can be displayed in theOHMD, optionally with different view perspectives adjusted for the leftand the right eye of the user 251 and 252. Left eye and right eyeoffsets or parallax can optionally be adjusted based on the distancefrom the OHMD, surgeon head or surgeon eyes to the surgical site using,for example, depth sensors or spatial mapping or other registrationtechniques and also based on interocular distance 253. Polarization orcolor techniques for stereoscopic views 254 can be combined withelectronic holograms such as those provided by the Microsoft Hololens.

In an alternative description in FIG. 16B, multiple 3D models 260, 261,262 can be generated, e.g. one for subsurface anatomic or pathologicstructures of the patient, one for virtual surgical tools or instrumentsand one for virtual surgical implant components. These can beregistered, e.g. in a common coordinate system or multiple coordinatesystems using coordinate transfers, also with the OHMD 263. Using sharedcoordinates for the different virtual 3D models 260, 261, 262 multipleviewers using multiple OHMDs can share a 3D World 264 with projection ordisplay of one or more of the models onto the live data of the patient265. The display can be generated separately for the left eye of eachuser using the user's left eye coordinates 266 and the right eye of eachuser using the user's right eye coordinates 267.

Stereoscopic views or different perspective views or views with aparallax for the left eye and the right eye can be generated formultiple virtual data sets or data volumes of the patient. Any of thedimensions listed in Table 4 or virtual structures, tissues or datamentioned in the application can be displayed separately for the lefteye and the right eye using stereoscopic views or different perspectiveviews or views with a parallax, simultaneously, non-simultaneously, orsequentially. In addition, any of the virtual data in Table 11 can bedisplayed using stereoscopic views or different perspective views orviews with a parallax for the left eye and the right eye. Multiple ofthe data listed in Table 11 can be displayed simultaneously,non-simultaneously or sequentially, for example also with the live dataor images of the patient seen through the OHMD, stereoscopically ornon-stereoscopically:

TABLE 11: Exemplary, non-limiting list of virtual data of the patient,surgical sites and alterations to surgical sites, surgical instrumentsand surgical steps or procedures, and medical devices that can bedisplayed, optionally simultaneously, using stereoscopic views ordifferent perspective views or views with a parallax for the left eyeand the right eye or non-stereoscopically. Virtual data are typicallydisplayed in conjunction with viewing or displaying live data of thepatient. Virtual data can be displayed stereoscopically ornon-stereoscopically or combinations thereof if multiple virtual datasets are displayed in the OHMD.

TABLE 11A: Exemplary virtual data of the patient that can be displayedstereoscopically or non-stereoscopically

-   -   Native anatomy, e.g.        -   Gyri of the brain        -   Venous sinus of the brain        -   Arterial structures of the brain        -   Brain lesion        -   Brain tumor        -   Features of the face        -   Features of an ear        -   Liver margin        -   Liver lobes        -   Spleen margin        -   Kidney, renal outline        -   One or more osteophytes        -   Bone spurs        -   Bony anatomy        -   Bony deformity        -   Acetabular rim of a hip        -   Tri-radiate cartilage region        -   Fovea capitis        -   Anterior superior iliac spine        -   Anterior inferior iliac spine        -   Symphysis pubis        -   Femoral head of a hip        -   Femoral neck        -   Greater trochanter        -   Lesser trochanter        -   Condyles of a knee        -   Trochlea of a knee        -   Patella of a knee        -   Tibial plateau of a knee        -   Medial tibial plateau of a knee        -   Lateral tibial plateau of a knee        -   Anterior cruciate ligament of a knee        -   Posterior cruciate ligament of a knee        -   Distal tibia of an ankle joint        -   Distal fibula of an ankle joint        -   Talus of an ankle joint        -   Any ligament or ligamentous structure of a patient        -   Glenoid rim of a shoulder        -   Glenoid of a shoulder        -   Humeral head or neck of a shoulder        -   Facet joint of a spine        -   Spinous process        -   Pedicle of a spine        -   Vertebral endplate        -   Intervertebral disk        -   Herniated disk        -   Any tumor affecting the human body

TABLE 11B: Exemplary virtual surgical sites and alterations to asurgical site that can be displayed stereoscopically ornon-stereoscopically

-   -   Alterations planned to surgical site, e.g.        -   Tissue removal        -   Removal of normal tissue        -   Removal of diseased tissue        -   Removal of neoplastic tissue        -   Bone cuts        -   Reaming (e.g. in proximal femur)        -   Broaching (e.g. in proximal femur)        -   Impacting (e.g. in a femur or a tibia)        -   Milling        -   Drilling        -   Tissue transplants        -   Organ transplants        -   Partial or complete resections, e.g. of organs        -   Placement of a medical device        -   Placement of a stent

TABLE 11C: Exemplary virtual surgical instruments and surgical steps orprocedures that can be displayed stereoscopically ornon-stereoscopically

-   -   Tissue cutters,        -   e.g. scalpels, blades, drills, saws, burrs, reamers,            broaches    -   Tissue ablation devices        -   e.g. heat or cryotherapy    -   Robotic arms    -   Instruments attached to robotic arms    -   Endoscopy devices    -   Endoscopic cameras    -   Endoscopic cutting devices    -   Endoscopic ablation devices    -   A predetermined surgical path or predetermined placement or        position, location, rotation, orientation, alignment, or        direction of one surgical instrument    -   A predetermined surgical path or predetermined placement or        position, location, rotation, orientation, alignment, or        direction of more than one surgical instrument    -   A predetermined surgical path or predetermined placement or        position, location, rotation, orientation, alignment, or        direction of more than one surgical instrument used        simultaneously    -   A predetermined surgical path or predetermined placement or        position, location, rotation, orientation, alignment, or        direction of more than one surgical instrument used        non-simultaneously    -   A predetermined surgical path or predetermined placement or        position, location, rotation, orientation, alignment, or        direction of more than one surgical instrument used in        succession    -   A predetermined surgical path or predetermined placement or        position, location, rotation, orientation, alignment, or        direction of more than one surgical instrument not used in        succession    -   A predetermined surgical path or predetermined placement or        position, location, rotation, orientation, alignment, or        direction of more than one surgical instrument used on the same        side of a joint    -   A predetermined surgical path or predetermined placement or        position, location, rotation, orientation, alignment, or        direction of more than one surgical instrument used on one or        more opposing sides of a joint    -   A predetermined surgical path or predetermined placement or        position, location, rotation, orientation, alignment, or        direction of more than one surgical instrument used on the same        vertebral levels    -   A predetermined surgical path or predetermined placement or        position, location, rotation, orientation, alignment, or        direction of more than one surgical instrument used on adjacent        vertebral levels    -   A predetermined surgical path or predetermined placement or        position, location, rotation, orientation, alignment, or        direction of more than one surgical instrument used on        non-adjacent vertebral levels    -   A predetermined surgical path or predetermined placement or        position, location, rotation, orientation, alignment, or        direction of one surgical instrument used on a vertebral        endplate    -   A predetermined surgical path or predetermined placement or        position, location, rotation, orientation, alignment, or        direction of more than one surgical instrument used on a        superior vertebral endplate and on an adjacent, inferior        vertebral endplate    -   A predetermined surgical path or predetermined placement or        position, location, rotation, orientation, alignment, or        direction of an instrument used for disk removal

TABLE 11D: Exemplary virtual medical devices and implants that can bedisplayed stereoscopically or non-stereoscopically

-   -   Hip replacement components        -   Acetabular cup including predetermined placement or            position, location, rotation, orientation, alignment,            anteversion, retroversion, inclination, offset, location in            relationship to the safe zone        -   Acetabular liner including predetermined placement or            position, location, rotation, orientation, alignment,            anteversion, retroversion, inclination, offset, location in            relationship to the safe zone        -   Femoral head including predetermined placement or position,            location, rotation, orientation, alignment, anteversion,            retroversion, inclination, offset, location in relationship            to the safe zone        -   Femoral neck including predetermined placement or position,            location, rotation, orientation, alignment, anteversion,            retroversion, inclination, offset, location in relationship            to the safe zone (optionally with modular necks)        -   Femoral stem including predetermined placement or position,            location, rotation, orientation, alignment, anteversion,            retroversion, inclination, offset, location in relationship            to the femoral neck cut, the calcar, the greater or the            lesser trochanter, the acetabulum    -   Knee replacement components        -   Femoral component including predetermined placement or            position, location, internal or external rotation,            orientation, alignment, flexion, extension, position in            relationship to anterior cortex, or mechanical axis or other            axis alignment, all optionally through the range of motion        -   Tibial component including predetermined placement or            position, location, internal or external rotation,            orientation, alignment, flexion, extension, slope, position            in relationship to cortical rim, or mechanical axis or other            axis alignment, all optionally through the range of motion        -   Polyethylene or other inserts including predetermined            placement or position, location, internal or external            rotation, orientation, alignment, flexion, extension, slope,            position in relationship to cortical rim, or mechanical axis            or other axis alignment, all optionally through the range of            motion        -   Patellar component including predetermined placement or            position, location, internal or external rotation,            orientation, alignment, position in relationship to patellar            cortical rim, position in relationship to trochlea,            optionally in flexion and/or extension and/or through the            range of motion, position in relationship to mechanical            axis, trochlear axis, trochlear groove, epicondylar axis or            other axis alignment        -   Trial femoral component including predetermined placement or            position, location, internal or external rotation,            orientation, alignment, flexion, extension, position in            relationship to anterior cortex, or mechanical axis or other            axis alignment, all optionally through the range of motion        -   Trial tibial component including predetermined placement or            position, location, internal or external rotation,            orientation, alignment, flexion, extension, slope, position            in relationship to cortical rim, or mechanical axis or other            axis alignment, all optionally through the range of motion        -   Trial inserts including predetermined placement or position,            location, internal or external rotation, orientation,            alignment, flexion, extension, slope, position in            relationship to cortical rim, or mechanical axis or other            axis alignment, all optionally through the range of motion        -   Trial patellar component including predetermined placement            or position, location, internal or external rotation,            orientation, alignment, position in relationship to patellar            cortical rim, position in relationship to trochlea,            optionally in flexion and/or extension and/or through the            range of motion, position in relationship to mechanical            axis, trochlear axis, trochlear groove, epicondylar axis or            other axis alignment    -   Spinal screws including predetermined placement or position,        location, rotation, orientation, alignment, location in        relationship to the pedicle, the cortical bone of the pedicle,        the endosteal bone of the pedicle, the posterior cortical bone        of the vertebral body, the anterior cortical bone of the        vertebral body, the lateral cortical bone of the vertebral body,        the superior endplate, the inferior endplate, the intervertebral        disk, the vertebral body, the trabecular bone of the vertebral        body, any fracture components or fragments, e.g. involving a        pedicle, a facet joint or a vertebral body    -   Pedicle screws including predetermined placement or position,        location, rotation, orientation, alignment, location in        relationship to the pedicle, the cortical bone of the pedicle,        the endosteal bone of the pedicle, the posterior cortical bone        of the vertebral body, the anterior cortical bone of the        vertebral body, the lateral cortical bone of the vertebral body,        the superior endplate, the inferior endplate, the intervertebral        disk, the vertebral body, the trabecular bone of the vertebral        body, any fracture components or fragments, e.g. involving a        pedicle, a facet joint or a vertebral body    -   Spinal rods including predetermined placement or position,        location, rotation, orientation, alignment, location in        relationship to one or more pedicles, the cortical bone of the        pedicle, the posterior cortical bone of the vertebral body, the        anterior cortical bone of the vertebral body, the lateral        cortical bone of the vertebral body, the superior endplate, the        inferior endplate, the intervertebral disk, the vertebral body,        any fracture components or fragments, e.g. involving a pedicle,        a facet joint or a vertebral body, a scoliotic deformity, and        predetermined correction for a scoliotic deformity    -   Artificial spinal disks including predetermined placement or        position, location, rotation, orientation, alignment, location        in relationship to one or more pedicles, the cortical bone of        the pedicle, the posterior cortical bone of the vertebral body,        the anterior cortical bone of the vertebral body, the lateral        cortical bone of the vertebral body, the superior endplate, the        inferior endplate, the intervertebral disk, the vertebral body,        any fracture components or fragments, e.g. involving a pedicle,        a facet joint or a vertebral body, a scoliotic deformity, and        predetermined correction for a scoliotic deformity    -   Metal screws, pins, plates, rods for trauma including        predetermined placement or position, location, rotation,        orientation, alignment, location in relationship to one or more        pedicles, the cortical bone of the pedicle, the posterior        cortical bone of the vertebral body, the anterior cortical bone        of the vertebral body, the lateral cortical bone of the        vertebral body, the superior endplate, the inferior endplate,        the intervertebral disk, the vertebral body, any fracture        components or fragments, e.g. involving a pedicle, a facet joint        or a vertebral body, a long bone, a joint, an articular surface,        and any predetermined correction fora fracture or fracture        deformity    -   Intramedullary nails including predetermined placement or        position, location, rotation, orientation, alignment, location        in relationship to one or more fracture components or fragments,        e.g. a long bone, a joint, an articular surface, and any        predetermined correction for a fracture or fracture deformity    -   Vascular stents        -   Coronary stents including predetermined placement or            position, location, rotation, orientation, alignment, for            example in relationship to an area of stenosis, an area of            vascular occlusion, a thrombus, a clot, a plaque, an ostium,            two or more ostia, an aneurysm, a dissection, an intimal            flap, adjacent vessels, adjacent nerves        -   Carotid stents including predetermined placement or            position, location, rotation, orientation, alignment, for            example in relationship to an area of stenosis, an area of            vascular occlusion, a thrombus, a clot, a plaque, an ostium,            two or more ostia, an aneurysm, a dissection, an intimal            flap, adjacent vessels, adjacent nerves        -   Aortic stents including predetermined placement or position,            location, rotation, orientation, alignment, for example in            relationship to an area of stenosis, an area of vascular            occlusion, a thrombus, a clot, a plaque, an ostium, two or            more ostia, an aneurysm, a dissection, an intimal flap,            adjacent vessels, adjacent nerves        -   Femoral stents including predetermined placement or            position, location, rotation, orientation, alignment, for            example in relationship to an area of stenosis, an area of            vascular occlusion, a thrombus, a clot, a plaque, an ostium,            two or more ostia, an aneurysm, a dissection, an intimal            flap, adjacent vessels, adjacent nerves    -   Cochlear implants including predetermined placement or position,        location, rotation, orientation, alignment, for example in        relationship to osseous structures, neural structures, auditory        structures, the labyrinth    -   Retinal implants including predetermined placement or position,        location, rotation, orientation, alignment, for example in        relationship to osseous structures, neural structures, vascular        structures    -   Neural implants including predetermined placement or position,        location, rotation, orientation, alignment, for example in        relationship to neural structures, vascular structures, osseous        structures    -   Neuroprosthetics including predetermined placement or position,        location, rotation, orientation, alignment, for example in        relationship to neural structures, vascular structures, osseous        structures    -   Implants for deep brain stimulation, e.g. for treatment of        Parkinson's disease including predetermined placement or        position, location, rotation, orientation, alignment, for        example in relationship to neural structures, vascular        structures, osseous structures

The list in Table 11 is only exemplary and is not meant to be limitingof the invention. Any of the exemplary virtual data of the patientlisted in Table 11A, exemplary virtual surgical sites and alterations toa surgical site listed in Table 11B, exemplary virtual surgicalinstruments and surgical steps or procedures listed in Table 11C, andexemplary virtual medical devices and implants listed in Table 11D canbe displayed by the OHMD in two, three or more dimensions (e.g. asdescribed also in Table 4), using stereoscopic as well asnon-stereoscopic projections or view. Thus, the invention is not limitedto stereoscopic displays and/or 2D displays and/or 3D displays. Anycombination of virtual displays is possible, e.g. 3D stereoscopicpatient anatomy or surgical site with 2D surgical instrument displaysand/or 2D medical device displays, or 3D patient anatomy, with 3Dnon-stereoscopic surgical instrument display and/or 3D stereoscopicmedical device display.

Aligning or Superimposing Physical Surgical Instruments with VirtualSurgical Instruments

With virtual displays of the surgical instruments in the OHMD, thesurgical instruments displayed in the virtual data can be representativeof the physical surgical instruments used in the live patient and canhave the same projected dimensions and shape as the physical surgicalinstruments. As indicated in Table 11, the virtual view of the virtualsurgical instrument or instruments can, for example, indicate thepredetermined position, location, rotation, orientation, alignment,direction of a surgical instrument. When the physical surgicalinstrument is aligned with and/or superimposed onto the virtualrepresentation of the virtual surgical instrument, the surgical step canoptionally be executed or the surgeon can elect to make adjustments tothe position, location, rotation, orientation, alignment, direction of aphysical surgical instrument relative to the virtual surgicalinstrument, for example on the basis of a ligament tension or ligamentbalance, e.g. in flexion or extension. The resultant alteration of thelive surgical site induced by the surgical step in the live patient istypically consistent with the virtual surgical plan, when the virtualand physical surgical instruments are superimposed in their respectiveposition, location, rotation, orientation, alignment, or direction.

More than one surgical step can be executed in this manner, e.g. byaligning the physical surgical instruments with the correspondingvirtual surgical instruments using stereoscopic or non-stereoscopicdisplays of virtual surgical instruments. The aligning can be performedin two dimensions, three dimensions, and more than three dimensions. Thealigning can be performed with stereoscopic and non-stereoscopicdisplays. More than one virtual surgical step can be planned utilizingthe virtual surgical plan. Two or more virtual surgical steps can beplanned. The virtual surgical steps can include the major surgical stepsof the intended procedure, they can include optionally sub-steps, or,optionally, the entire procedure. When the physical surgical steps areexecuted after aligning one or more physical instruments with thevirtual instruments in the corresponding surgical steps, each surgicalstep using the physical instruments is effectively image guided using,optionally, the virtual surgical plan with the operator or the surgeonusing the image guidance information, for example from a preoperativescan or imaging study obtained at a time different from the surgicalprocedure, typically preceding the surgical procedure, and typicallywith the surgical site in a different object coordinate system at thetime of the preoperative imaging when compared to the time of thesurgical procedure. The display of the virtual surgical instruments canbe stereoscopic or non-stereoscopic.

Thus, by aligning physical surgical instruments seen through the OHMD ordisplayed by the OHMD with virtual surgical instruments usingstereoscopic or non-stereoscopic displays of virtual surgicalinstruments in the OHMD, it is possible to execute accurately on asurgical plan in the live patient using pre-existing image informationand image guidance information, as defined, for example, in a virtualsurgical plan. In addition, by aligning physical surgical instrumentsseen through the OHMD or displayed by the OHMD with virtual surgicalinstruments using stereoscopic or non-stereoscopic displays of virtualsurgical instruments in the OHMD, it is possible to achieve anpredetermined position, location, rotation, orientation, alignment,direction of a medical implant including, but not limited to, forexample the implants listed in Table 11D.

The OHMD can show the one or more virtual surgical instruments with acontinuous surface view, for example, using one color or multiple colorsfor different features of the instrument. The continuous surface displaycan include shading based on light sources used in the operating roomand/or over the surgical field. The directional orientation of the ORlight sources can, for example, be measured using image capture,optionally integrated into, attached to or separate from the OHMD.

The OHMD can show the one or more virtual surgical instruments with anoutline view which can be in 2D or in 3D. The outline view can includean outline of the entire virtual surgical instrument, for example in aparticular plane or cross-sectional plane. The outline view canoptionally only highlight select features of the virtual surgicalinstrument, e.g. a bone cutting surface or feature or a grip feature orcombinations thereof. The OHMD can show two or more outline views, forexample extending through or along the surface or the periphery of thevirtual surgical instrument along different planes. These planes can bechosen to be different than at a 0 or 180 degree angles to each other.In some embodiments of the invention, the outline views can beorthogonal to each other. In this manner, even though the two or moreoutline views can be two-dimensional, the OHMD can still provideinformation to the surgeon or the operator on the intended orientation,position and/or direction of the surgical instrument in three-dimensionsby providing two or more outline views with different angularorientations and by providing information on the x, y and z-axisalignment or position or orientation or direction of the surgicalinstrument. Outline views can help limiting the amount of informationdisplayed by the OHMD, which can help the surgeon maintaining his or herfocus on the surgical site, with full visibility of the surgical site.Outline view can help decrease the risk of obscuring important liveinformation from the patient, e.g. a bleeding vessel, by inadvertentlysuperimposing virtual data, e.g. 3D surface data, and obscuring portionsof the live anatomy.

By aligning physical surgical instruments seen through the OHMD ordisplayed by the OHMD with virtual surgical instruments usingstereoscopic or non-stereoscopic displays of virtual surgicalinstruments in the OHMD, it is possible to achieve certain alterationsof a surgical site or certain implant placement or implant componentplacement in live patients that can, for example, determine at least oneof a

-   -   Surgical instrument position    -   Surgical instrument location    -   Surgical instrument orientation    -   Surgical instrument rotation    -   Surgical instrument alignment    -   Surgical instrument direction    -   Depth of advancement of a surgical instrument, e.g. for        acetabular or glenoid reaming    -   Implant position    -   Implant location    -   Implant orientation    -   Implant rotation    -   Implant alignment    -   Implant position of two or more implant components in        relationship to each other and/or in relationship to the patient    -   Implant location of two or more implant components in        relationship to each other and/or in relationship to the patient    -   Implant orientation of two or more implant components in        relationship to each other and/or in relationship to the patient    -   Implant rotation of two or more implant components in        relationship to each other and/or in relationship to the patient    -   Implant alignment of two or more implant components in        relationship to each other and/or in relationship to the patient

Anatomic or pathologic structures and/or tissue including but notlimited to one or more osteophytes or bone spurs or other bony anatomyor deformity or soft-tissue or neoplastic tissue or abnormality can beused for referencing the patient both in the virtual and in the livedata and for determining or cross-referencing to the other anatomy thedesired instrument or implant component position, location, orientation,rotation or alignment.

Aligning or Superimposing Physical Surgical Instruments or PhysicalMedical Devices with Virtual Alterations to a Surgical Site

The OHMD can display virtual alterations to a surgical site superimposedonto the live surgical site prior to the physical alteration of the livesurgical site. The virtual alterations to a surgical can be simulatedusing a virtual surgical plan. The virtual surgical alterations and/orthe virtual surgical plan can be executed or displayed in two, three ormore dimensions, optionally with a stereoscopic or non-stereoscopicdisplay.

In some embodiments of the invention, the OHMD can display a virtualalteration to a surgical site. The operator or the surgeon can thenalign the physical surgical instrument selected to perform the intendedalteration to the physical surgical site and align the physical surgicalinstrument with the virtual alteration of the surgical site. The virtualalteration can, for example, be the removal or shape modification of oneor more osteophytes or bone spurs or other bony anatomy or deformity orsoft-tissue or neoplastic tissue or abnormality. The operator or surgeoncan then advance or move the physical surgical instrument into thedirection of or into the physical surgical site, optionally whilemaintaining alignment of the physical instrument with the virtualalteration of the surgical site. In this manner, the operator or thesurgeon can effect the desired change or alteration to the surgical sitein the live patient, and the change or alteration achieved in thesurgical site of the live patient is typically similar to or alignedwith or consistent with the intended virtual change or alteration to thesurgical site and, if applicable, the virtual surgical plan.

For example, a surgeon can plan to make a bone cut to a distal femur ofa patient. The OHMD can display the virtual bone cut superimposed ontothe uncut bone of the live patient. The virtual bone cut and theintended physical bone cut can, for example, remove or correct one ormore osteophytes or bone spurs or other bony anatomy or deformity orsoft-tissue. The surgeon can then align the saw blade of the physicalbone saw with the planar surface of the intended bone cut in the virtualalteration of the bone surface displayed by the OHMD. By advancing thesaw blade in the direction of the cut while maintaining alignmentbetween the physical saw blade, e.g. the flat surface of the physicalsaw blade, and the planar surface of the virtual bone cut, the surgeoncan achieve an accurate physical bone cut in the live patient.Alternatively, the surgeon can align a cutting tool or cut block or cutguide to guide a bone saw with the planar surface of the intended bonecut in the virtual alteration of the bone surface displayed by the OHMD;the cutting tool or cut block or cut guide can then optionally beaffixed to the tissue and/or bone, for example using one or more pins orscrews and the cut can be performed using the cutting tool, cut block orcut guide.

In another example, a surgeon can plan to make a bone cut to a proximalfemur of a patient, e.g. for partial or total hip arthroplasty, or to adistal femur or proximal tibia, e.g. for partial or total kneereplacement, or to a proximal humerus, e.g. for partial or totalshoulder arthroplasty. The OHMD can display the virtual bone cutsuperimposed onto the uncut bone of the live patient. The surgeon canthen align the saw blade of the physical bone saw with the planarsurface of the intended bone cut in the virtual alteration of the bonedisplayed by the OHMD. By advancing the saw blade in the direction ofthe cut while maintaining alignment between the physical saw blade andthe planar surface of the virtual bone cut, the surgeon can achieve anaccurate physical bone cut in the live patient. The bone cut can beoriented to achieve a desired component rotation and/or componentflexion or extension. The bone cut can be oriented to achieve a desiredslope. The same result can be achieved by aligning a cutting tool, cutblock, or cut guide with the planar surface of the virtual bone cut,optionally affixing it to the tissue and/or bone, and performing the cutwith the bone saw.

In another example, a surgeon can plan to ream or broach a bone, e.g. aproximal femur or a proximal humerus. The OHMD can display the boneafter the virtual reaming or broaching procedure showing the intendedvirtual alteration of the inner bone surface after the reaming orbroaching procedure; the display can optionally be superimposed onto thelive image of the unaltered physical bone. The surgeon can then alignthe physical reamer or broach onto the intended virtual alteration andshape change of the bone after the reaming or broaching proceduredisplayed by the OHMD. By advancing the reamer or broach in thedirection of the virtually reamed or broached bone surface whilemaintaining alignment between the physical reamer or broach and thevirtually reamed or broached bone surface, the surgeon can achieve anaccurate physical reaming or broaching of the bone in the live patient.

In another example, a surgeon can plan to place a pedicle screw in apedicle of a patient, e.g. for spinal fusion. The OHMD can display thevirtual bone void or space created by a virtual pedicle screw,optionally superimposed onto the unaltered pedicle of the live patient.The surgeon can then align a physical drill or a physical pedicle screwwith the virtual bone void or space for the pedicle screw in the virtualalteration of the pedicle displayed by the OHMD. By advancing thephysical drill or pedicle screw in the direction of the virtual bonevoid or space in the pedicle while maintaining alignment between thephysical drill or pedicle screw and the virtual bone void or space inthe pedicle, the surgeon can achieve accurate placement of the physicaldrill or pedicle screw in the live patient. The bone void in the pedicleor the position of the pedicle screw can be chosen in the virtualsurgical plan so that there is one or more desired minimum distance or aminimum area or volume of bone between the bone void or the pediclescrew and the endosteal bone surface or cortical bone surface of thepedicle, medially, laterally, superiorly, and/or inferiorly.

In another example, a surgeon can plan to place an intervertebral diskreplacement in an intervertebral disk space of a patient, e.g. formotion preserving disk replacement. The OHMD can display the virtualalteration required for the placement of the disk replacement, forexample with virtual alterations to the superior and/or inferiorendplates of the two adjacent vertebral bodies, optionally superimposedonto the endplates of the live patient. The virtual and intendedphysical alterations can include, for example, the removal of one ormore osteophytes or bone spurs or other bony anatomy or deformity or theresection of portions of or all of the endplate(s). The surgeon can thenalign physical instruments used for altering the vertebral endplates toaccept the intervertebral disk replacement with the virtual alterationof the endplates displayed by the OHMD. By advancing the physicalsurgical instruments in the direction of the virtual alteration of theendplates while optionally maintaining alignment between the physicalsurgical instrument and the virtual alteration of the endplates, thesurgeon can achieve accurate placement of the physical surgicalinstruments and the physical disk replacement in the live patient.

Thus, by aligning with or directing physical surgical instruments ormedical devices towards a display of virtual alterations to a surgicalsite in the OHMD it is possible to achieve certain alterations of asurgical site or certain implant placement or implant componentplacement in live patient that can, for example, determine at least oneof a

-   -   Surgical instrument position    -   Surgical instrument location    -   Surgical instrument orientation    -   Surgical instrument rotation    -   Surgical instrument alignment    -   Surgical instrument direction    -   Depth of advancement of a surgical instrument, e.g. for        acetabular reaming    -   Implant position    -   Implant location    -   Implant orientation, e.g. anteversion, retroversion, offset        (e.g. in a hip replacement acetabular cup or femoral component),        abduction, adduction, internal rotation, external rotation,        flexion, extension (e.g. in a knee replacement femoral component        or tibial component)    -   Implant rotation    -   Implant alignment    -   Implant position of two or more implant components in        relationship to each other and/or in relationship to the patient    -   Implant location of two or more implant components in        relationship to each other and/or in relationship to the patient    -   Implant orientation of two or more implant components in        relationship to each other and/or in relationship to the patient    -   Implant rotation of two or more implant components in        relationship to each other and/or in relationship to the patient    -   Implant alignment of two or more implant components in        relationship to each other and/or in relationship to the patient

Optionally, the surgeon can toggle the display of the virtual databetween a display of the surgical site prior to the alteration and/orafter the alteration. Optionally, the surgeon can advance the display ofthe virtual data several surgical steps so that, for example, not thenext but one or more subsequent virtual alterations to the surgical sitebe displayed.

Optionally, the surgeon can use displays with different colors forsimultaneously or non-simultaneously viewing the physical, live surgicalsite and the virtual surgical site before and after one or moreconsecutive or non-consecutive virtual alterations intended or plannedfor the surgical site, optionally superimposed onto the live or virtualsurgical site before the one or more alterations are made.

Optionally, the virtual display of the planned alteration can besuperimposed onto the physical surgical site after the surgicalalteration has been made to check for the accuracy of the physicalalteration in the live patient. If the surgeon notices a discrepancybetween the planned virtual alteration and the physical alteration, thesurgeon can modify the physical alteration. For example, if the surgeonhas executed a bone cut, for example in a proximal femur for a hipreplacement or in a distal femur or proximal tibia for a kneereplacement, the surgeon can use the OHMD to superimpose the planned,intended virtual bone cut onto the physical bone cut after the bone cutwas made. If the surgeon notices that the physical bone cut took lessbone than intended when compared to the planned, intended virtual bonecut, the surgeon can recut the bone to more closely match the physicalbone cut with the intended virtual bone cut and, optionally the virtualsurgical plan.

If the surgeon notices a discrepancy between the planned virtualalteration and the physical physical alteration, the surgeon canoptionally also modify the virtual alteration to match the physicalalteration induced by the patient. The virtual surgical plan can then bemodified, for example for one or more of the subsequent surgical stepsor procedures so that the virtual surgical plan will continue to workwith the physical surgical alterations achieved with or induced in thelive patient. The modification of the virtual surgical plan can beperformed manually by the operator or surgeon, semi automatically orautomatically using the input from the physical surgical alterationinduced in the patient.

For example, if the surgeon has executed a bone cut, e.g. in a proximalfemur for a hip replacement or in a distal femur or proximal tibia for aknee replacement, the surgeon can use the OHMD to superimpose theplanned, intended virtual bone cut onto the physical bone cut after thebone cut was made. If the surgeon notices that the physical bone cuttook more bone than intended when compared to the planned, intendedvirtual bone cut, the surgeon can modify the virtual surgical plan. Themodified surgical plan can then, for example, included that a subsequentbone cut or reaming step on the opposite articular surface will takeless bone, typically the same amount less bone on the opposite articularsurface than was removed too much during the prior physical bone cut inthe live patient. Alternatively, the modified surgical plan can includethat one or more components of the medical device be thicker tocompensate for the larger bone cut. In a knee replacement, for example,a thicker tibial insert can optionally be used. In a hip replacement,for example, a thicker acetabular liner or an offset liner canoptionally be used.

Aligning Physical Medical Devices and Implants with Virtual MedicalDevices and Implants

By aligning with or directing physical medical devices or medical devicecomponents towards a display of virtually implanted medical devices ormedical device components, for example in their intended final virtualposition, location, orientation, rotation or alignment, in the OHMD, itis possible to achieve predetermined implant placement or implantcomponent placement in the live patient that can, for example, determineat least one of a physical, final

-   -   Implant position    -   Implant location    -   Implant orientation, e.g. anteversion, retroversion, offset        (e.g. in a hip replacement acetabular cup or femoral component),        internal rotation, external rotation, flexion, extension (e.g.        in a knee replacement femoral component or tibial component)    -   Implant rotation    -   Implant alignment    -   Implant position of two or more implant components in        relationship to each other and/or in relationship to the patient    -   Implant location of two or more implant components in        relationship to each other and/or in relationship to the patient    -   Implant orientation of two or more implant components in        relationship to each other and/or in relationship to the patient    -   Implant rotation of two or more implant components in        relationship to each other and/or in relationship to the patient    -   Implant alignment of two or more implant components in        relationship to each other and/or in relationship to the patient

The OHMD can show the one or more virtual and, optionally, virtuallyimplanted medical devices or medical device components with a continuoussurface view, for example, using one color or with multiple colors fordifferent features of the device or for different device components. Thecontinuous surface display can include shading based on light sourcesused in the operating room and/or over the surgical field. Thedirectional orientation of the OR light sources can, for example, bemeasured using image capture, optionally integrated into, attached to orseparate from the OHMD.

The OHMD can show the one or more virtual and, optionally, virtuallyimplanted medical devices or medical device components with an outlineview which can be in 2D or in 3D. The outline view can include anoutline of the entire virtual medical device or virtual medical devicecomponent, for example in a particular plane or cross-sectional plane.The outline view can optionally only highlight select features of thevirtual medical device or virtual medical device component, e.g. a bonefacing surface or a surface between two or more components facing eachother, or a linking portion of the device or component or combinationsthereof. The OHMD can show two or more outline views, for exampleextending through or along the surface or the periphery of the virtualmedical device or virtual medical device component along differentplanes. These planes can be chosen to be different than at a 0 or 180degree angle to each other. In some embodiments of the invention, theoutline views can be orthogonal to each other. In this manner, eventhough the two or more outline views can be two-dimensional, the OHMDcan still provide information to the surgeon or the operator on theintended orientation, position and/or direction of the device or devicecomponent in three-dimensions by providing two or more outline viewswith different angular orientations and by providing information on thex, y and z-axis alignment or position or orientation of the device ordevice component. Outline views can help limiting the amount ofinformation displayed by the OHMD, which can help the surgeonmaintaining his or her focus on the surgical site, with full visibilityof the surgical site. Outline view can help decrease the risk ofobscuring important live information from the patient, e.g. an exposednerve root, by superimposing virtual data in a reduced format.

Optionally, the surgeon can toggle the display of the virtual databetween a display of one or more of the virtual medical devicecomponents and, optionally, the live medical device components.

Optionally, the surgeon can use displays with different colors forsimultaneously or non-simultaneously viewing the two or more virtualmedical device components, optionally superimposed onto or displayedwith the physical medical device.

Optionally, the virtual display of the medical device or medical devicecomponent after virtual implantation can be superimposed onto thephysical medical device or medical device component after the physicalimplantation or placement to check for the accuracy of the physicalimplantation or placement in the live patient. If the surgeon notices adiscrepancy between the planned virtual position, location, orientation,rotation, alignment of the medical device or medical device componentsand the physical position, location, orientation, rotation, alignment ofthe physical medical device or medical device components, the surgeoncan modify the physical device placement or the surgeon can utilizedifferent device components, e.g. in a knee replacement use a thicker ora thinner or a differently shaped tibial polyethylene insert or in a hipreplacement use a different polyethylene liner, e.g. thicker, thinner orwith offsets.

Visors

In some embodiments of the invention, a visor or splash shield can beintegrated into the OHMD to protect the surgeon including his or hereyes from bodily fluids, e.g. blood. In some embodiments of theinvention, a visor or splash shield can be attached to the OHMD toprotect the surgeon including his or her eyes from bodily fluids, e.g.blood. In some embodiments of the invention, a visor or splash shieldcan be placed in front of the OHMD to protect the surgeon including hisor her eyes from bodily fluids, e.g. blood.

Color Coding

Optionally, the different surgical instruments, devices or devicecomponents can be color coded during the display in the OHMD. Forexample, the color coding in the OHMD display will correspond to thecolor coding of the physical surgical instruments, devices or devicecomponents, if applicable. An exemplary color coding chart is providedbelow:

Physical Device:

4.0 mm screw—grey

4.5 mm screw—pink

5.0 mm screw—brown

5.5 mm screw—blue

6.0 mm screw—orange

6.5 mm screw—yellow

7.0 mm screw—no color

7.5 mm screw—green

8.5 mm screw—black

Virtual Device Display:

4.0 mm screw—grey

4.5 mm screw—pink

5.0 mm screw—brown

5.5 mm screw—blue

6.0 mm screw—orange

6.5 mm screw—yellow

7.0 mm screw—no color

7.5 mm screw—green

8.5 mm screw—black

Such screws can, for example, be used with pedicle screws or glenoidcomponents or acetabular components. The foregoing color coding is onlyexemplary. Any colors, combination of colors, stripes, patterns can beused for identifying different sizes, dimensions, shapes, diameters,widths or lengths. Any instrument or implant can be color coded.

Color coding is applicable to any surgical instrument, medical device ormedical device component, e.g. also with vascular stents, cardiacimplants, cardiac defibrillators, hip replacement components, kneereplacement components etc.

Optionally, in addition to the color coding or as an alternative tocolor coding, the OHMD can display one or more numerical values next tothe virtual surgical instrument or medical device, e.g. a thickness ordiameter or a size from a sizing chart.

In some embodiments of the invention, the OHMD can recognize if there isa discrepancy in diameter, width, length, dimension, shape, or size ofan physical surgical instrument or device and a virtual device chosen ina surgical plan. For example, an image and/or video capture systemintegrated into, attached to or connected to the OHMD or separate fromthe OHMD can be used to image a surgical instrument, medical device ormedical device component, optionally correct its diameter, width,length, dimension, shape, or size based on the distance of the surgicalinstrument or device from the image and/or video capture system (e.g.using parallax based measurements or registration or calibrationphantoms) and then determine if the physical medical device or medicaldevice component chosen by the operator or surgeon matches that selectedin the virtual surgical plan. If the physical surgical instrument ormedical device or medical device component is mismatched, for examplewith regard to diameter, width, length, dimension, shape, or sizerelative to the virtual instrument or component, the system can providea warning signal, such as an acoustic alert or a visual warning sign(e.g. a red exclamation mark displayed by the OHMD).

Partially Visible or Partially Obscured Instruments, Tools, Devices,Implants, Implant

Components In certain situations during surgery or in certain surgicalsites, one or more physical surgical instruments or tools or one or morephysical devices, implants, implant components and systems forimplantation may only be partially visible during aspects or a period ofthe surgery. This is particular the case with surgeries involving deepseated organs, e.g. a liver or a kidney, a brain, or deep seated,obscured or hidden body structures, e.g. a hip joint or aspects of aspine, where important parts of one or more physical surgicalinstruments or tools or one or more physical devices, implants, implantcomponents and systems for implantation may be at least partiallyobscured from view. This may be aggravated if the portion that isobscured from view is a portion that is inducing one or more alterationto a tissue surface, for example by electro-cautery, ablation, cuttingor reaming or impacting. This reduction or limitation in visualizationof the one or more physical surgical instruments or tools or one or morephysical devices, implants, implant components and systems forimplantation can result in a decreased accuracy of the surgicaltechnique and, for example, placement errors of a device, implant,implant component or system for implantation or potential complications.In an embodiment of the invention, one or more of the physical surgicalinstruments or tools and/or one or more of the physical devices,implants, implant components and systems for implantation can includecertain standardized geometric features, e.g. rectangles, triangles,circles and the like, that can be readily recognized by an image and/orvideo capture system integrated into or attached to or coupled to orseparate from the OHMD. Alternatively, the image and/or video capturesystem may simply recognize the visible geometric shapes, surfaces,features or portions of the one or more of the physical surgicalinstruments or tools and/or one or more of the physical devices,implants, implant components and systems for implantation. Theinformation can then be used to compute the shape, geometry, outline,surface or other features of the non-visualized, non-visible portions ofthe one or more of the physical surgical instruments or tools and/or oneor more of the physical devices, implants, implant components andsystems for implantation. With any of the foregoing techniques, theposition, location, orientation, alignment, motional direction, and/ortrajectory of the one or more of the surgical instruments or toolsand/or one or more of the devices, implants, implant components andsystems for implantation can be determined even though the one or moreof the surgical instruments or tools and/or one or more of the devices,implants, implant components and systems for implantation is onlypartially or incompletely visualized or visible in the surgical site.

The non-visualized or non-visible portions of the one or more of thephysical surgical instruments or tools and/or one or more of thephysical devices, implants, implant components and systems forimplantation can then optionally be displayed by the OHMD and projectedonto the view of the surgical site. Optionally, the non-visualized ornon-visible portions of the one or more of the physical surgicalinstruments or tools and/or one or more of the physical devices,implants, implant components and systems for implantation can bedisplayed by the OHMD simultaneous with the one or more of thecorresponding virtual surgical instruments or tools and/or one or moreof the corresponding virtual devices, implants, implant components andsystems for implantation. Different colors or display patterns canoptionally be used to display and differentiate the virtual from thephysical of the one or more of the surgical instruments or tools and/orone or more of the devices, implants, implant components and systems forimplantation in the OHMD display.

In alternative embodiments, one or more of the physical surgicalinstruments or tools and/or one or more of the physical devices,implants, implant components and systems for implantation can includeone or more IMUs, including, for example, with accelerometers,magnetometers, and gyroscopes, similar, for example, to the OHMD. Insome embodiments, one or more of the physical surgical instruments ortools and/or one or more of the physical devices, implants, implantcomponents and systems for implantation can include one or moreradiofrequency tags or markers or retroreflective markers and the likeand its/their position, location and/or orientation can be captured by asurgical navigation system. Optionally, the OHMD may also include one ormore radiofrequency tags or markers or retroreflective markers and thelike and its position, location and/or orientation can also be capturedby the surgical navigation system and cross-referenced to the one ormore of the physical surgical instruments or tools and/or one or more ofthe physical devices, implants, implant components and systems forimplantation. One or more of the physical surgical instruments or toolsand/or one or more of the physical devices, implants, implant componentsand systems for implantation can also include light sources, such aslasers or LEDs. A laser can be projected, for example, on a wall or aceiling and the OHMD and the patient can be referenced in relationshipto that. An LED attached to or integrated into the one or more of thephysical surgical instruments or tools and/or one or more of thephysical devices, implants, implant components and systems forimplantation can be recognized, for example, by an image and/or videocapture system integrated into or attached to or coupled to or separatefrom the OHMD.

With any of the foregoing techniques, the position, location,orientation, alignment, motional direction, and/or trajectory of the oneor more of the physical surgical instruments or tools and/or one or moreof the physical devices, implants, implant components and systems forimplantation can be determined even though the one or more of thephysical surgical instruments or tools and/or one or more of thephysical devices, implants, implant components and systems forimplantation is only partially or incompletely visualized or visible inthe surgical site. A computer program or software can then optionallycompute the shape, geometry, outline, surface of other features of thenon-visualized, non-visible portions of the one or more of the physicalsurgical instruments or tools and/or one or more of the physicaldevices, implants, implant components and systems for implantation. Thenon-visualized or non-visible portions of the one or more of thephysical surgical instruments or tools and/or one or more of thephysical devices, implants, implant components and systems forimplantation can then optionally be displayed by the OHMD and projectedonto the view of the surgical site.

Optionally, the non-visualized or non-visible portions of the one ormore of the physical surgical instruments or tools and/or one or more ofthe physical devices, implants, implant components and systems forimplantation can be displayed by the OHMD simultaneous with the one ormore of the corresponding virtual surgical instruments or tools and/orone or more of the corresponding virtual devices, implants, implantcomponents and systems for implantation. Different colors or displaypatterns can optionally be used to display and differentiate the virtualfrom the physical of the one or more of the surgical instruments ortools and/or one or more of the devices, implants, implant componentsand systems for implantation in the OHMD display.

Difficult Lighting and Tissue Contrast Conditions

In certain situations during surgery or in certain surgical sites, thelighting conditions and tissue contrast may be such that any virtualanatomic data or structures, virtual surgical plans, virtual tool orinstrument or device paths, virtual surgical instruments or tools and/orany virtual devices, implants, implant components and systems forimplantation may be difficult to see in the OHMD display by the humanoperator. In any of these circumstances, the system can optionally allowthe operator or the surgeon to change the display mode or it canactively change the display mode of one or more the virtual anatomicdata or structures, virtual surgical plans, virtual tool or instrumentor device paths, virtual surgical instruments or tools and/or thevirtual devices, implants, implant components and systems forimplantation, for example by changing the color, brightness, intensity,and/or contrast of one or more of the virtual anatomic data orstructures, virtual surgical plans, virtual tool or instrument or devicepaths, virtual surgical instruments or tools and/or the virtual devices,implants, implant components and systems for implantation. Differentchanges in color, brightness, intensity, and/or contrast can be appliedto different virtual data, e.g. virtual anatomic data or structures,virtual surgical plans, virtual tool or instrument or device paths,virtual surgical instruments or tools and/or the virtual devices,implants, implant components and systems for implantation.

The surgeon or operator or the software or the system may change thecolor of one or more of the virtual anatomic data or structures, virtualsurgical plans, virtual tool or instrument or device paths, virtualsurgical instruments or tools and/or the virtual devices, implants,implant components and systems for implantation. The surgeon or operatoror the software or the system may change the brightness of one or moreof the virtual anatomic data or structures, virtual surgical plans,virtual tool or instrument or device paths, virtual surgical instrumentsor tools and/or the virtual devices, implants, implant components andsystems for implantation. The surgeon or operator or the software or thesystem may change the intensity of one or more of the virtual anatomicdata or structures, virtual surgical plans, virtual tool or instrumentor device paths, virtual surgical instruments or tools and/or thevirtual devices, implants, implant components and systems forimplantation. The surgeon or operator or the software or the system maychange the contrast of one or more of the virtual anatomic data orstructures, virtual surgical plans, virtual tool or instrument or devicepaths, virtual surgical instruments or tools and/or the virtual devices,implants, implant components and systems for implantation. The surgeonor the operator or the software or the system may change the displaypattern of the one or more of the virtual anatomic data or structures,virtual surgical plans, virtual tool or instrument or device paths,virtual surgical instruments or tools and/or the virtual devices,implants, implant components and systems for implantation. For example,one or more of the virtual anatomic data or structures, virtual surgicalplans, virtual tool or instrument or device paths, virtual surgicalinstruments or tools and/or the virtual devices, implants, implantcomponents and systems for implantation may be displayed with a rasterpattern or a line pattern or a point pattern or any other displaypattern known in the art. Alternatively, one or more of the virtualanatomic data or structures, virtual surgical plans, virtual tool orinstrument or device paths, virtual surgical instruments or tools and/orthe virtual devices, implants, implant components and systems forimplantation may be displayed with a temporally changing displaypattern, including, but not limited to a blinking pattern or a flashingpattern, e.g. with only intermittent display of the virtual information.Alternatively, one or more of the virtual anatomic data or structures,virtual surgical plans, virtual tool or instrument or device paths,virtual surgical instruments or tools and/or the virtual devices,implants, implant components and systems for implantation may bedisplayed with a “skeletonization pattern”, wherein, for example, onlykey features or key outlines of one or more of the virtual anatomic dataor structures, virtual surgical plans, virtual tool or instrument ordevice paths, virtual surgical instruments or tools and/or the virtualdevices, implants, implant components and systems for implantation canbe displayed. Alternatively, one or more of the virtual anatomic data orstructures, virtual surgical plans, virtual tool or instrument or devicepaths, virtual surgical instruments or tools and/or the virtual devices,implants, implant components and systems for implantation may bedisplayed with a “highlighting pattern” or mode, wherein, for example,key features or key outlines of one or more of the virtual anatomic dataor structures, virtual surgical plans, virtual tool or instrument ordevice paths, virtual surgical instruments or tools and/or the virtualdevices, implants, implant components and systems for implantation maybe displayed using an enlargement of the feature or outline or a coloror brightness or contrast or other display enhancement of the feature oroutline. Optionally, less important features or outline components orportions may be reduced in display intensity or removed from thedisplay. The foregoing display adjustments can be performed via operatorcontrolled commands, e.g. manual or voice or other commands.Alternatively, these adjustments can be semi-automatic with operatorinput or automatic using, for example, information about brightness,contrast and/or color of the virtual and/or the live data of the patientas well as ambient light conditions, e.g. OR light intensity, lightreflections, etc. For semi-automatic or automated adjustment of thedisplay of select, one or more virtual data, e.g. virtual anatomic dataor structures, virtual surgical plans, virtual tool or instrument ordevice paths, virtual surgical instruments or tools and/or the virtualdevices, implants, implant components and systems for implantation,light intensity and contrast sensors can be employed which canoptionally be integrated into, attached to or separate from one or moreOHMDs. Alternatively, the information about color, brightness,intensity, contrast of the live data seen through the OHMD and/orambient lighting conditions can be obtained through one or more imageand/or video capture systems integrated into, attached to or separatefrom the OHMD.

Any of the foregoing changes to the display of virtual anatomic data orstructures, virtual surgical plans, virtual tool or instrument or devicepaths, surgical instruments or tools and/or the devices, implants,implant components and systems for implantation can also be applied toany partially obscured or non-visible portions of the physical surgicalinstruments or tools and/or the physical devices, implants, implantcomponents and systems for implantation.

Any of the virtual anatomic data or structures, virtual surgical plans,virtual tool or instrument or device paths, surgical instruments ortools and/or the devices, implants, implant components and systems forimplantation described anywhere in the invention can be modified in thedisplay using one or more of these techniques or any other technique ofdisplay modification known in the art.

In certain situations during surgery or in certain surgical sites, thelighting conditions and tissue contrast may be such that any obscuredportions of the anatomy or obscured pathology or obscured targettissue(s) or deep seated, obscured or hidden portions of the anatomy ortarget tissue(s) or intended alterations to deep seated tissue(s) may bedifficult to see in the OHMD display by the human operator. Thisincludes also normal tissue and normal anatomic structures, hidden orobscured or deep seated. In any of these circumstances, the system canoptionally allow the operator or the surgeon to change the display modeor the system can actively change the display mode of the anatomy ordeep seated portions of the anatomy or target tissue(s) or intendedalterations to deep seated, obscured or hidden tissue(s). For example,the surgeon or operator may change the color of the anatomy or deepseated, obscured or hidden portions of the anatomy or target tissue(s)or intended alterations to deep seated, obscured or hidden tissue(s).Alternatively, the surgeon or the operator may change the displaypattern of the anatomy or deep seated, obscured or hidden portions ofthe anatomy or target tissue(s) or intended alterations to deep seated,obscured or hidden tissue(s). For example, the anatomy or deep seated,obscured or hidden portions of the anatomy or target tissue(s) orintended alterations to deep seated, obscured or hidden tissue(s) may bedisplayed with a raster pattern or a line pattern or a point pattern orany other display pattern known in the art. Alternatively, the anatomyor deep seated, obscured or hidden portions of the anatomy or targettissue(s) or intended alterations to deep seated, obscured or hiddentissue(s) may be displayed with a temporally changing display pattern,including, but not limited to a blinking pattern or a flashing pattern,e.g. with only intermittent display of the information. Alternatively,the anatomy or deep seated, obscured or hidden portions of the anatomyor target tissue(s) or intended alterations to deep seated, obscured orhidden tissue(s) may be displayed with a “skeletonization pattern”,wherein, for example, only key features or key outlines of the anatomyor deep seated, obscured or hidden portions of the anatomy or targettissue(s) or intended alterations to deep seated, obscured or hiddentissue(s) may be displayed. Alternatively, the anatomy or deep seated,obscured or hidden portions of the anatomy or target tissue(s) orintended alterations to deep seated, obscured or hidden tissue(s) may bedisplayed with a “highlighting pattern” or mode, wherein, for example,key features or key outlines of the anatomy or deep seated, obscured orhidden portions of the anatomy or target tissue(s) or intendedalterations to deep seated, obscured or hidden tissue(s) may bedisplayed using an enlargement of the feature or outline or a color orbrightness or contrast or other display enhancement of the feature oroutline. Optionally, less important features or outline components orportions may be reduced in display intensity or removed from thedisplay. Any of the tissues described anywhere in the invention, such asby way of example, a cerebral cortex, gyri, a pedicle, vertebralendplates, an anterior vertebral wall, a posterior vertebral wall, anacetabulum, vessels, nerves, tumors, can be modified in the displayusing one or more of these techniques or any other method of displaymodification known in the art.

Any of the foregoing adjustments in color, brightness, intensity, and/orcontrast can be applied to 2D or 3D, stereoscopic and non-stereoscopicdisplays of one or more of the virtual anatomic data or structures,virtual surgical plans, virtual tool or instrument or device paths,virtual surgical instruments or tools and/or the virtual devices,implants, implant components and systems for implantation. If live dataof the patient are not directly seen through the OHMD, but are capturedthrough an image and/or video capture system integrated into, attachedto or separate from the OHMD and then displayed by the OHMD, optionallyin combination with virtual anatomic data or structures, virtualsurgical plans, virtual tool or instrument or device paths, virtualsurgical instruments or tools and/or the virtual devices, implants,implant components and systems for implantation, the same or similaradjustments can be applied to one or more of the live data of thepatient, e.g. select anatomic structures, or all of the live data of thepatient.

In some aspects, the invention provides a method for preparing a jointfor a prosthesis in a patient. In some embodiments, the method comprisesregistering the patient's joint and one or more optical head mounteddisplays worn by a surgeon or surgical assistant in a coordinate system,obtaining one or more intra-operative measurements, registering the oneor more intra-operative measurements in the coordinate system,developing a virtual surgical plan based on the one or moreintra-operative measurements, and displaying or projecting aspects ofthe virtual surgical plan superimposed onto the corresponding portionsof the patient's joint with the optical head mounted display. In someembodiments, the one or more optical head mounted displays areregistered in the same coordinate system. In some embodiments, the oneor more intra-operative measurements are morphological measurements,optical measurements or combinations thereof. In some embodiments, theone or more intra-operative measurements are not pressure measurements.

In some aspects, the invention provides a method for preparing anorthopedic procedure in a patient. In some embodiments, the methodcomprises registering the patient's surgical site and one or moreoptical head mounted display worn by a surgeon or surgical assistant ina common coordinate system, obtaining one or more intra-operativeoptical measurements using one or more optical markers, registering theone or more intra-operative optical measurements using one or moreoptical markers in the common coordinate system, developing a virtualsurgical plan based on the one or more intra-operative opticalmeasurements, and displaying or projecting aspects of the virtualsurgical plan superimposed onto the corresponding portions of thepatient's physical joint with the optical head mounted display. Thevirtual surgical plan can be displayed or projected onto the patient'sphysical joint based at least in part on coordinates of thepredetermined position of the virtual surgical plan.

In some embodiments, the virtual surgical plan incorporates data from apre-operative scan. In some embodiments, the virtual surgical planincorporates data from an intra-operative scan. In some embodiments, thevirtual surgical plan incorporates data from a pre-operative scan and anintra-operative scan. The scan includes one or more x-rays, a CT scan,an MRI scan, an ultrasound or combinations thereof.

In some embodiments, the scan data are registered in the commoncoordinate system. In some embodiments, the registered scan data aredisplayed superimposed onto the surgical site by the optical headmounted display. In some embodiments, the scan data include athree-dimensional display of the surgical site.

In some embodiments, the registering step includes identifying one ormore landmarks in the live surgical site. In some embodiments, one ormore corresponding anatomic landmarks are identified in the patient'sscan data.

In some embodiments, the registering step includes identifying one ormore anatomic axes or biomechanical axes in the live surgical site. Insome embodiments, the one or more corresponding anatomic axes orbiomechanical axes are identified in the patient's scan data.

In some embodiments, the live surgical site includes one or more of abone, a cartilage, a joint, a joint surface, an opposing joint surface,a ligament, a meniscus, a labrum, an intra-articular structure, aspinous process, a pedicle, a facet joint, a superior or inferiorprocess or a vertebral body.

In some embodiments, the registering step includes detecting one or moreoptical markers attached to one or more structures in the live surgicalsite. In some embodiments, the registering step includes detecting oneor more optical markers attached to the OR table. In some embodiments,the detecting of the one or more optical markers includes determiningone or more of a position, orientation, alignment, direction of movementor speed of movement of the one or more optical markers.

The optical marker can include a geometric pattern, a QR code, a barcodeor combinations thereof. The QR code or barcode can be included in orintegrated into or attached to the geometric pattern.

In some embodiments, the optical head mounted display includes one ormore cameras or image capture or video capture systems. The one or morecameras or image capture or video capture systems can detect the one ormore optical markers including their coordinates (x, y, z). In someembodiments, the optical marker includes information about implantinventory management. For example, the QR code can include informationabout implant inventory management.

In some embodiments, the one or more cameras or image capture or videocapture systems included in the optical head mounted display reads theinventory management in the QR and transmits it to another computer.

In some embodiments, the intraoperative measurement includes identifyingcoordinates (x, y, z) of a live anatomic landmark in the patient's jointusing one or more optical markers. In some embodiments, theintraoperative measurement includes identifying coordinates (x, y, z) ofan anatomic landmark in the intra-operative scan data.

In some embodiments, the one or more optical markers are radiopaque andtheir coordinates (x, y, z) can be detected in the intra-operative scandata.

In some embodiments, the optical markers are detected using the one ormore cameras or image capture or video capture systems included in theoptical head mounted display and detected in the intra-operative scandata are registered in the common coordinate system.

In some embodiments, the intraoperative measurement includes identifyingan anatomic axis or a biomechanical axis of the patient. Thebiomechanical axis can be a mechanical axis of the leg. In someembodiments, the intraoperative measurement includes obtaininginformation from a surgically altered surface.

In some embodiments, the intraoperative measurement includes identifyinga center of rotation of a joint of the patient. The joint can be thejoint being operated on, or the joint can be a joint different than thejoint being operated on.

In some embodiments, the intraoperative measurement includes identifyingan anatomic plane. The anatomic plane can be tangent with one or moreanatomic landmarks. The anatomic plane can intersect one or moreanatomic landmarks. In some embodiments, the anatomic plane can be foundby placing a virtual plane to be tangent with or intersect with one ormore anatomic landmarks. The virtual plane can be placed using a virtualinterface.

In some embodiments, the virtual surgical plan includes predeterminedpath for a surgical instrument. In some embodiments, the virtualsurgical plan includes a projected or intended cut plane. In someembodiments, the virtual surgical plan includes a virtual cut blockprojected in a desired or intended position, orientation and/oralignment. In some embodiments, the virtual surgical plan includes aprojected or intended reaming, milling or impacting axis. In someembodiments, the virtual surgical plan includes a virtual surgicalinstrument displayed or projected in a desired or predeterminedposition, orientation, alignment and/or direction of movement. In someembodiments, the virtual surgical plan includes a virtual surgicalimplant component displayed or projected in a desired or predeterminedposition, orientation and/or alignment.

In some aspects, the method of preparing a joint for a prosthesis in apatient comprises obtaining scan data associated with the joint of thepatient; preparing a virtual surgical plan for the patient's joint basedon the scan data; registering the patient's physical joint, the virtualsurgical plan, and one or more optical head mounted displays worn by asurgeon or surgical assistant in a common coordinate system, obtainingone or more intra-operative measurements, adjusting or modifying thevirtual surgical plan based on the one or more intra-operativemeasurements, and displaying or projecting aspects of the adjusted ormodified surgical plan superimposed onto corresponding portions of thepatient's physical joint with the optical head mounted display. In someembodiments, the one or more intra-operative measurements aremorphological measurements, optical measurements or combinationsthereof.

In some aspects, the method of preparing an orthopedic procedure in apatient comprises obtaining scan data associated with the surgical siteof the patient; preparing a virtual surgical plan for the patient basedon the scan data; registering the patient's live surgical site, thevirtual surgical plan, and one or more optical head mounted displaysworn by a surgeon or surgical assistant in a common coordinate system,obtaining one or more intra-operative measurements; adjusting ormodifying the virtual surgical plan based on the one or moreintra-operative measurements, and displaying or projecting aspects ofthe adjusted or modified surgical plan superimposed onto correspondingportions of the patient's live surgical site with the optical headmounted display.

In some embodiments, the one or more intra-operative measurementsinclude one or more optical markers.

In some embodiments, the scan data is obtained pre-operatively and/orintra-operatively. In some embodiments, the scan data includepre-operative and intra-operative scan data. In some embodiments, thescan data include one or more x-rays, a CT scan, an MRI scan, anultrasound or combinations of the foregoing.

In some embodiments, the scan data are registered in the commoncoordinate system. In some embodiments, the registered scan data aredisplayed superimposed onto the surgical site by the optical headmounted display.

In some embodiments, the scan data include a three-dimensional displayof the surgical site.

In some embodiments, the registering includes identifying one or moreanatomic landmarks in the patient's scan data. In some embodiments, theregistering includes identifying one or more corresponding landmarks inthe live surgical site. In some embodiments, the registering includesidentifying one or more anatomic axes or biomechanical axes in thepatient's scan data. In some embodiments, the registering includesidentifying one or more corresponding anatomic axes or biomechanicalaxes in the live surgical site.

In some embodiments, the live surgical site includes one or more of abone, a cartilage, a joint, a joint surface, an opposing joint surface,a ligament, a meniscus, a labrum, an intra-articular structure, aspinous process, a pedicle, a facet joint, a superior or inferiorprocess or a vertebral body.

In some embodiments, the registering includes detecting one or moreoptical markers attached to one or more structures in the live surgicalsite. In some embodiments, the registering includes detecting one ormore optical markers attached to the OR table. In some embodiments, thedetecting of the one or more optical markers includes determining one ormore of a position, orientation, alignment, direction of movement orspeed of movement of the one or more optical markers.

The optical marker can include a geometric pattern, a QR code, a barcodeor combinations thereof. The QR code or barcode can be included in orintegrated into or attached to the geometric pattern.

In some embodiments, the optical head mounted display includes one ormore cameras or image capture or video capture systems. The one or morecameras or image capture or video capture systems can detect the one ormore optical markers including their coordinates (x, y, z). In someembodiments, the optical marker includes information about implantinventory management. For example, the QR code can include informationabout implant inventory management.

In some embodiments, the QR code includes information about implantinventory management. In some embodiments, the one or more cameras orimage capture or video capture systems included in the optical headmounted display reads the inventory management in the QR and transmitsit to another computer.

In some embodiments, the intraoperative measurement includes identifyingcoordinates (x, y, z) of a live anatomic landmark in the patient's jointusing one or more optical markers. In some embodiments, theintraoperative measurement includes identifying coordinates (x, y, z) ofan anatomic landmark in the intra-operative scan data.

In some embodiments, the one or more optical markers are radiopaque andtheir coordinates (x, y, z) can be detected in the intra-operative scandata.

In some embodiments, the optical markers are detected using the one ormore cameras or image capture or video capture systems included in theoptical head mounted display and detected in the intra-operative scandata are registered in the common coordinate system.

In some embodiments, the intraoperative measurement includes identifyingan anatomic axis or a biomechanical axis of the patient. For example,the biomechanical axis can be a mechanical axis of the leg.

In some embodiments, the intraoperative measurement includes identifyinga center of rotation of a joint of the patient. The joint can be thejoint being operated on or a joint different than the joint beingoperated on.

In some embodiments, the intraoperative measurement includes identifyingan anatomic plane. The anatomic plane can be tangent with one or moreanatomic landmarks. The anatomic plane can intersect one or moreanatomic landmarks. In some embodiments, the anatomic plane is found byplacing a virtual plane to be tangent with or intersect with one or moreanatomic landmarks. The virtual plane can be placed using a virtualinterface.

In some embodiments, the intraoperative measurement includes obtaininginformation from a surgically altered surface.

In some embodiments, the adjusting or modifying the virtual surgicalplan includes placing or moving a predetermined path for a surgicalinstrument. In some embodiments, the adjusting or modifying the virtualsurgical plan includes the placing or moving of a virtual cut plane. Insome embodiments, the adjusting or modifying the virtual surgical planincludes the placing or moving of a virtual cut block. In someembodiments, the adjusting or modifying the virtual surgical planincludes the placing or moving of a virtual reaming, milling orimpacting axis. In some embodiments, the adjusting or modifying thevirtual surgical plan includes the placing or moving of a virtualsurgical instrument. In some embodiments, the adjusting or modifying thevirtual surgical plan includes the placing or moving of a virtualsurgical implant component. According to some aspects of the invention,the method of preparing a joint for a prosthesis in a patient comprisesregistering the patient's live surgical site and one or more opticalhead mounted displays worn by a surgeon or surgical assistant in acommon coordinate system, obtaining one or more intra-operativemeasurements, registering the one or more intra-operative measurementsin the common coordinate system, developing a virtual surgical planbased on the one or more intra-operative measurements, the virtualsurgical plan including at least one virtual cut plane, and displayingor projecting the one or more virtual cut planes superimposed onto thecorresponding portions of the patient's live surgical site with theoptical head mounted display.

According to some aspects of the invention, the method of preparing ajoint for a prosthesis in a patient comprises registering the patient'slive surgical site and an optical head mounted display worn by a surgeonor surgical assistant in a common coordinate system, developing avirtual surgical plan, registering the virtual surgical plan in thecommon coordinate system, the virtual surgical plan including at leastone virtual cut plane, and displaying or projecting the at least onevirtual cut planes superimposed onto the corresponding portions of thepatient's live surgical site with the optical head mounted display. Insome embodiments, the method further comprises obtaining one or moreintra-operative measurements. In some embodiments, the method furthercomprises registering the one or more intra-operative measurements inthe common coordinate system. In some embodiments, the one or moreintra-operative measurements comprise intra-operative morphological andoptical measurements.

In some embodiments, the prosthesis is a knee replacement and thevirtual cut plane defines a tibial slope after implantation of thetibial implant component(s). In some embodiments, the prosthesis is aknee replacement and the virtual cut plane defines an angle of varus orvalgus correction in relationship to the patient's mechanical axis ofthe leg for a tibial component and related bone cuts. In someembodiments, the prosthesis is a knee replacement and the virtual cutplane defines an angle of varus or valgus correction in relationship tothe patient's mechanical axis of the leg for a femoral component andrelated bone cuts. In some embodiments, the prosthesis is a kneereplacement and the virtual cut planes define an angle of varus orvalgus correction in relationship to the patient's mechanical axis ofthe leg for a femoral component and a tibial component and related bonecuts including a combined correction. In some embodiments, theprosthesis is a knee replacement and the virtual cut plane correspondsto a distal femoral cut and defines a femoral component flexion. In someembodiments, the prosthesis is a knee replacement and the virtual cutplane corresponds to an anterior femoral cut and defines a femoralcomponent rotation. In some embodiments, the prosthesis is a kneereplacement and the virtual cut plane corresponds to a posterior femoralcut and defines a femoral component rotation. In some embodiments, theprosthesis is a knee replacement and the virtual cut plane correspondsto chamfer cut and defines a femoral component rotation. In someembodiments, the prosthesis is a hip replacement and wherein the virtualcut plane defines a leg length after implantation.

According to some aspects of the invention, the method for preparing ajoint for a prosthesis in a patient comprises registering the patient'slive surgical site and one or more optical head mounted displays worn bya surgeon or surgical assistant in a common coordinate system,developing a virtual surgical plan, registering the virtual surgicalplan in the common coordinate system, the virtual surgical planincluding at least two or more projected or intended pin or drill paths,and displaying or projecting the two or more projected or intended pinor drill paths superimposed onto the corresponding portions of thepatient's bone or cartilage in the live surgical site with the opticalhead mounted display.

In some embodiments, a first physical pin or drill is aligned with thefirst virtual pin or drill path and the pinning or drilling is executedwhile maintaining the alignment. In some embodiments, a second physicalpin or drill is aligned with the second virtual pin or drill path andthe pinning or drilling is executed while maintaining the alignment. Insome embodiments, the first and second pins or drills are used to fixateor reference a surgical guide or cut block. In some embodiments, thedrill holes created by the physical first and second pins or drills areused to fixate or reference a surgical guide or cut block.

In some embodiments, the surgical guide or cut block is used to executea bone cut. The bone cut can define a leg length based on the virtualsurgical plan. The bone cut can define a varus or valgus correctionbased on the virtual surgical plan. The bone cut can define a femoralcomponent flexion based on the virtual surgical plan. The bone cut candefine a femoral component rotation based on the virtual surgical plan.The bone cut can determine a tibial slope based on the virtual surgicalplan. In some embodiments, the bone cut is a keel punch and determines atibial component rotation based on the virtual surgical plan.

According to some aspects, the method for preparing a joint for aprosthesis in a patient comprises registering the patient's joint andone or more optical head mounted displays worn by a surgeon or surgicalassistant in a common coordinate system, obtaining one or moreintra-operative measurements, registering the one or moreintra-operative measurements in the common coordinate system, developinga virtual surgical plan based on the one or more intra-operative[morphological or optical] measurements, the virtual surgical planincluding a virtual surgical drill guide, and displaying or projectingthe virtual drill guide superimposed onto the corresponding portions ofthe patient's live surgical site intended for the drilling with theoptical head mounted display. In some embodiments, the one or moreintra-operative measurements are morphological and/or opticalmeasurements. In some embodiments, the physical drill corresponding tothe virtual drill guide includes at least two openings to accommodatetwo or more drills. In some embodiments the virtual drill guidecorresponds to a physical drill guide and has at least one or moredimensions similar to the physical drill guide. According to someaspects, the method for preparing a joint for a prosthesis in a patientcomprises registering the patient's joint and one or more optical headmounted displays worn by a surgeon or surgical assistant in a commoncoordinate system, obtaining one or more intra-operative measurements,registering the one or more intra-operative measurements in the commoncoordinate system, developing a virtual surgical plan based on the oneor more intra-operative measurements, the virtual surgical planincluding at least one virtual axis for a reamer, a mill or an impactor,and displaying or projecting the at least one virtual axis for a reamer,a mill or an impactor superimposed onto the corresponding portions ofthe patient's live surgical site with the optical head mounted display.

According to some aspects, the method for preparing a joint for aprosthesis in a patient comprises registering the patient's joint and anoptical head mounted display worn by a surgeon or surgical assistant ina common coordinate system, developing a virtual surgical plan,registering the virtual surgical plan in the common coordinate system,the virtual surgical plan including at least one virtual axis for areamer, a mill or an impactor, and displaying or projecting the at leastone virtual axis for a reamer, a mill or an impactor superimposed ontothe corresponding portions of the patient's live surgical site with theoptical head mounted display. In some embodiments, the method furthercomprises obtaining one or more intra-operative measurements. In someembodiments, the method further comprises registering the one or moreintra-operative measurements in the common coordinate system.

In some embodiments, the prosthesis is a hip replacement and the virtualaxis defines an acetabular anteversion after implantation of theacetabular component(s) based on the virtual surgical plan. In someembodiments, the prosthesis is a hip replacement and the virtual axisdefines an acetabular offset after implantation of the acetabularcomponent(s) based on the virtual surgical plan. In some embodiments,the prosthesis is a hip replacement and the virtual axis defines acombined acetabular and femoral component anteversion. A physicalreamer, mill or impactor can be aligned with the virtual axis for thereamer, mill or impactor and the reaming, milling or impacting can beexecuted while maintaining the alignment.

According to some aspects, the method for preparing a joint for aprosthesis in a patient comprises registering the patient's livesurgical site and one or more optical head mounted displays worn by asurgeon or surgical assistant in a common coordinate system, obtainingone or more intra-operative measurements, registering the one or moreintra-operative measurements in the common coordinate system, developinga virtual surgical plan based on the one or more intra-operativemeasurements, the virtual surgical plan including a virtual tibialtemplate, and displaying or projecting the virtual tibial templatesuperimposed onto the cut tibia with the optical head mounted display.

In some embodiments, the physical tibial template is aligned with thevirtual cut tibial template, a tibial keel punch is inserted, and theproximal tibia is punched to accommodate the tibial keel and fins.

In some embodiments, the virtual and the physical tibial templatedetermine the alignment and rotation of the tibial implant component.

According to some aspects, the method for preparing an orthopedicprocedure in a patient comprises registering the patient's surgical siteand one or more] optical head mounted displays worn by a surgeon orsurgical assistant in a common coordinate system, wherein theregistration of the patient's surgical site in the common coordinatesystem is performed using one or more optical markers attached to thepatient in or around the surgical site, wherein the optical markerincludes one or more geometric patterns, wherein the optical markers aredetected with a camera, an image capture or video system integratedinto, attached to or separate from the optical head mounted display. Insome embodiments, the optical marker includes at least one portion thatis radiopaque. In some embodiments, internal structures of the patientor the surgical site are visualized using an imaging test with ionizingradiation. For example, the imaging test can be one or more x-raysand/or a CT scan.

In some embodiments, the radiopaque portions of the optical marker aredetected on the imaging test using image processing software. In someembodiments, the radiopaque portions of the optical marker detected onthe imaging test are cross-referenced with the visible portions of theoptical marker detected with the camera, image capture or video systemand wherein the information is used to register the internal structuresof the patient or the surgical site in the common coordinate system.

In some embodiments, the optical head mounted display displays theinternal structures of the patient or the surgical site superimposedonto the corresponding external surfaces of the patient or the surgicalsite. In some embodiments, the optical head mounted display superimposesa virtual surgical plan onto the corresponding external and internalstructures. The virtual surgical plan can be a predetermined path for asurgical device.

Examples

The following examples show representative applications of variousembodiments of the invention. The examples are not meant to be limiting.Someone skilled in the art will recognize other applications ormodifications of the methods, techniques, devices and systems described.Any embodiment described for one joint or anatomic region, e.g. a spineor pedicle, can be applied to other joints or other regions, e.g. a hip,hip replacement, knee, knee replacement, vascular imaging study,angiography etc.

In some embodiments, when a physical guide, tool, instrument or implantis aligned with or superimposed onto a virtual surgical guide, tool,instrument or implant displayed or projected by the OHMD, the aligningor superimposing can be performed with a location accuracy of about 10mm, about 9 mm, about 8 mm, about 7 mm, about 6 mm, about 5 mm, about 4mm, about 3 mm, about 2 mm, about 1 mm, about 0.5 mm, about 0.25 mm, orless, 0.25 mm to 0.5 mm, 0.25 mm to 1 mm, 0.25 mm to 2 mm, 0.25 mm to 3mm, 0.25 mm to 4 mm, 0.25 mm to 5 mm, 0.25 mm to 6 mm, 0.25 mm to 7 mm,1 mm to 2 mm, 1 mm to 3 mm, 1 mm to 4 mm, 1 mm to 5 mm, 1 mm to 6 mm, 1mm to 7 mm, 2 mm to 3 mm, 2 mm to 4 mm, 2 mm to 5 mm, 2 mm to 6 mm, 2 mmto 7 mm, 3 mm to 4 mm, 3 mm to 5 mm, 3 mm to 6 mm, 3 mm to 7 mm, 4 mm to5 mm, 4 mm to 6 mm, 4 mm to 7 mm, 5 mm to 6 mm, 5 mm to 7 mm, 6 mm to 7mm or as needed depending on the clinical application, in one, two orthree directions, x, y, z. When the physical guide, tool, instrument orimplant is aligned with or superimposed onto the virtual surgical guide,tool, instrument or implant displayed or projected by the OHMD, thealigning or superimposing can be performed with an orientation or angleaccuracy of about 10°, about 9°, about 8°, about 7°, about 6°, about 5°,about 4°, about 3°, about 2°, about 1°, about 0.5°, about 0.25° or less,0.25-10°, 0.25 to 9°, 0.25-8°, 0.25-7°, 0.25-6°, 0.25-5°, 0.25-4°,0.25-3°, 0.25-2°, 0.25-1°, 0.25-0.5°, 0.5 to 9°, 0.5-8°, 0.5-7°, 0.5-6°,0.5-5°, 0.5-4°, 0.5-3°, 0.5-2°, 0.5-1°, 1 to 9°, 1-8°, 1-7°, 1-6°, 1-5°,1-4°, 1-3°, 1-2°, 2-9°, 2-8°, 2-7°, 2-6°, 2-5°, 2-4°, 2-3°, 3-9°, 3-8°,3-7°, 3-6°, 3-5°, 3-4°, 4-9°, 4-8°, 4-7°, 4-6°, 4-5°, 5-9°, 5-8°, 5-7°,5-6°, 6-9°, 6-8°, 6-7°, 7-9°, 7-8°, 8-9° or as needed depending on theclinical application, in one, two or three directions, x, y, z.

The mechanical axis of the lower extremity is determined by drawing aline from the center of the femoral head to the center of the anklejoint, which corresponds typically to an approximately 3° slope comparedwith that of the vertical axis. This can be subdivided into the femoralmechanical axis, which runs from the head of the femur to theintercondylar notch of the distal femur, and the tibial mechanical axis,which extends from the center of the proximal tibia to the center of theankle. The medial angle formed between the mechanical axis of the femurand the mechanical axis of the tibia is called the hip-knee-ankle angle,which represented the overall alignment of the lower extremity and isusually about or slightly less than 180° in normal knees, also callednormal mechanical axis alignment. The position of the mechanical axiscauses it to usually pass just medial to the tibial spine, but this canvary widely based on the patient height and pelvic width.

Pedicle Screw, Spinal Rod Placement for Example for Correction of SpinalDeformities, Scoliosis and/or Fracture Treatment

Pedicle screw and rod placement is one of the most common spinalprocedures. It can be performed for a number of different conditions,including, for example, spinal instability, correction of spinaldeformities, e.g. scoliosis, kyphosis and combinations thereof, as wellas congenital spinal defects. Pedicle screw and rod placement can becombined with bone graft, e.g. allograft or autograft. Sometimes,infusable or injectable bone morphogenic protein can be used during theprocedure to facilitate healing and stabilization of bone graft.

Preoperatively, patients will commonly undergo x-ray imaging, forexample in anteroposterior, lateral and oblique views. Special views ofselect regions, e.g. the sacrum or the occipito-atlantic junction can beobtained. X-rays can be obtained in standing and lying position. X-rayscan also be obtained in prone or supine position. X-rays may be obtainedwith the patient erect, spinal flexion and spinal extension. X-rays mayalso be obtained with the patient bending to the left side or to theright side.

Patients may optionally undergo CT scanning or MRI scanning. CT scanningand MRI scanning have the added advantage of providing a 3D dataset ofthe patient's anatomy. Moreover, the thecal sac and the nerve roots canbe visualized. With MRI, the spinal cord can also be visualized.

Virtual Surgical Plan

The surgeon can develop a virtual surgical plan for the pedicle screwand rod placement which can optionally incorporate any desired deformitycorrection. Typical criteria for placement of pedicle screws can includethe following:

The entry point of the pedicle screw and any awl, probe, tap, k-wire,y-wire, other wires, and other surgical instruments can be chosen, forexample, to be at the lateral border of the superior articular processwith the intersect to a horizontal line bisecting the transverseprocesses on the left and right side.

In the lumbar spine, the trajectory of the pedicles will typicallyconverge 5-10 degrees in the upper lumbar spine, 10-15 degrees in thelower lumbar spine. Typically no cephalad or caudad tilt of thetrajectory is needed in the lumbar spine.

In the thoracic spine, the entry point can be just below the rim of theupper facet joint, and approximately 3 mm lateral to the center of thejoint near the base of the transverse process. In the thoracic spine,the pedicles and with that the screws can converge to the midline atapproximately 7-10 degrees; in the sagittal plane, they can be oriented10-20 degrees caudally. In accessing T12, the virtual surgical plan caninclude removal of transverse process to open the marrow space. Theangulation can be medial and caudal angulation.

Surgeon can generally use between a lateral intersection method forpedicle screw placement, with the lateral border of the superiorarticular processes forming an intersect to a horizontal line bisectingthe transverse processes on the left and right side. A more medial entrypoint can be chosen, in which case a rangeur may be required to removethe base of the articular process. This can be included in the virtualsurgical plan.

For S1, the entry point can be chosen at the intersect of a verticalline tangential to the S1 articular process and a horizontal linetangential to its inferior border. Typically, at S1, pedicle screwsconverge, but an overhanging pelvis may limit this in vivo. The screwswill typically aim at the superior border of sacral promontory. Theinstrument placement and the pedicle screw placement in the virtualsurgical plan will be selected or defined in a manner where the pediclescrew and/or the instruments will avoid the S1 foramen and any nerveroots. If bicortical screws are used, the screw position will beselected or oriented in order to avoid any injury to the L5 nerve roots;any imaging test such as a CT scan or an MRI scan can be used toidentify the L5 nerve root and to place the pedicle screw(s) in thevirtual surgical plan, with optional display of the CT or MRI scan andthe nerve root, so that its tip and body have a safety margin relativeto the nerve root.

The virtual surgical plan can comprise a 2D or 3D display of the spinalstructures. The 2D display can be a multiplanar display, for exampleshowing the spine in axial, oblique axial, sagittal, oblique or curvedsagittal, coronal, oblique or curved coronal projections. A 3D displaycan show the spine, for example, from a posterior projection, ananterior projection, a lateral projection, a projection from the top orthe bottom, or a projection along a nerve root or the thecal sac or thecord. Representative bony structures that can be displayed in thismanner include, for example, the spinous processes, the lamina, thefacet joints, the pedicles and the vertebral bodies including theendplates, anterior, posterior, medial and lateral cortex. In someembodiments of the invention, the view perspective will be theperspective that the surgeon's head and the OHMD have relative to thesurgical field and the patient. The perspective can be different for theleft eye display and the right eye display, in particular whenstereoscopic display technique is used, with substantially identicalview angles of the virtual data of the patient seen by the surgeon'sleft eye through the display of the OHMD unit and the live data of thepatient seen by the surgeon's left eye through the OHMD unit andsubstantially identical view angles of the virtual data of the patientseen by the surgeon's right eye through the display of the OHMD unit andthe live data of the patient seen by the surgeon's right eye through theOHMD unit.

In some embodiments, the thecal sac, neural structures and nerve roots,e.g. L4, L5, and S1 are highlighted in the surgical plan in addition tothe bony structures. The nerve roots can be highlighted usingsegmentation techniques known in the art, e.g. automatic orsemi-automatic or manual segmentation. Alternatively, an operator or asurgeon can click on the nerve root in the vicinity of a pedicle orintended pedicle screw placement. The location of the click can bestored in the image data volume and can be highlighted with a differentcolor. The area or volume that includes the click can be registered as asafety zone which the pedicle screw and any instruments used for theplacement should not enter. A safety margin, e.g. of 2, 3, 4, 5, 7 or 10mm can be added to the safety zone. The surgical plan and the placementor position or orientation of any pedicle screw and relatedinstrumentation will be modified or adapted during the virtual planningto ensure that no nerve damage or impingement will be caused by thesurgical procedure.

In some embodiments, vascular structures can be highlighted usingautomated, semi-automated, or manual segmentation techniques or simpleclicks or image markings performed by a surgeon or operator. Suchvascular structures can, for example, include the aorta, the inferiorvena cava, any branches of the aorta or the inferior vena cava,intercostal arteries, the innominate artery. A safe zone and/or a safetymargin of 2, 3, 4, 5, 7 or 10 mm or more mm can be defined around thesevascular structures. The surgical plan and the placement or position ororientation of any pedicle screw and related instrumentation will bemodified or adapted during the virtual planning to ensure that novascular damage will be caused in the surgical procedure.

The virtual surgical plan can include

-   -   Identifying the desired pedicle screw position and/or location        and/or orientation    -   Identifying the desired position and/or location and/or        orientation and/or trajectory of any surgical instrument used        for placing the pedicle screw, e.g. an awl, a probe, a wire, a        tab, a screw driver and the like, including the pedicle screw        itself.    -   Identifying the desired rod position and/or location and/or        orientation    -   Identifying the desired spinal deformity correction if        applicable, e.g. correction of kyphosis, lordosis, scoliosis,        sagittal deformity, coronal deformity, rotational deformity,        facture deformity    -   Identifying sensitive structures, e.g. neural structures, nerve        roots, vascular structures    -   Defining safe zone, e.g. for cortical penetration, e.g. in a        pedicle, neural structures, nerve roots and/or vascular        structures

The virtual surgical plan can include, optionally predefined, criteriato automated or semi-automated virtual placement of a pedicle screw inthe patient's data. Such criteria can include the distance between thepedicle screw or related bone void to accept the pedicle screw to themedial, lateral, superior, and/or inferior endosteal surface or corticalsurface in portions or all of the pedicle or the area or volume betweenpedicle screw or related bone void to accept the pedicle screw to themedial, lateral, superior, and/or inferior endosteal surface or corticalsurface in portions or all of the pedicle. If the surgeon manually,visually places the virtual pedicle screw on the 2D or 3D display, thesame or similar criteria can be applied by the software to highlightpotential areas that may result in clinical problems, e.g. a corticalbreach or a nerve root injury. For example, if a virtual pedicle screwcomes within 1, 2, or 3 mm of the medial cortex of a pedicle, thesoftware, using image processing and segmentation of the bone, endostealbone or cortical bone, can highlight such proximity and potential risk.The highlighting can occur, for example, by color coding areas ofproximity to a cortex or to a neural or vascular structure or by othervisual cues and acoustic warning signals. Such highlighted areas canoptionally also be displayed by the OHMD during the surgical procedure,stereoscopically or non-stereoscopically. Optionally, highlighted areascan be displayed in outline format.

The selection of a size, width, diameter or length of a pedicle screwcan also be performed in a manual, semi-automatic or automatic matterusing criteria such as the distance between the pedicle screw or relatedbone void to accept the pedicle screw to the medial, lateral, superior,and/or inferior endosteal surface or cortical surface in portions or allof the pedicle or the area or volume between pedicle screw or relatedbone void to accept the pedicle screw to the medial, lateral, superior,and/or inferior endosteal surface or cortical surface in portions or allof the pedicle.

The surgeon can place the digital hologram of the virtual pedicle screwmanually, for example using a virtual interface, on the virtual displayof the patient's hidden subsurface anatomy using criteria such aslocation of the pedicle screw including its tip in the vertebral body,location of the pedicle screw including its tip in relationship to aspinal/vertebral body fracture, location of the pedicle screw includingits tip in relationship to a superior endplate, location of the pediclescrew including its tip in relationship to an inferior endplate,location of the pedicle screw including its tip in relationship to ananterior vertebral cortex and/or a posterior vertebral cortex, locationof the pedicle screw including its tip in relationship to a vessel,location of the pedicle screw including its tip in relationship to theaorta, location of the pedicle screw including its tip in relationshipto the inferior vena cava, location of the pedicle screw including itstip in relationship to neural structures, the thecal sac, nerve rootsand/or the spinal cord, distance, area or volume between the pediclescrew including its tip to a spinal/vertebral body fracture, distance,area or volume between the pedicle screw including its tip to a superiorendplate, distance, area or volume between of the pedicle screwincluding its tip to an inferior endplate, distance, area or volumebetween the pedicle screw including its tip to the an anterior and/orposterior vertebral cortex, distance, area or volume between the pediclescrew including its tip to a vessel, distance, area or volume betweenthe pedicle screw including its tip to the aorta, distance, area orvolume between the pedicle screw including its tip to the inferior venacava, distance, area or volume between the pedicle screw including itstip to neural structures, the thecal sac, nerve roots and/or the spinalcord. The surgeon can use this information on location or distance orarea or volume also to select the size, width, diameter or length of thepedicle screw in the virtual surgical plan or using the virtualrepresentations of the pedicle screw(s) and the patient's anatomy. Safezone criteria can be defined for the foregoing criteria, for example 1,2 or 3 or 5 or more mm from a cortex or a neural structure. If thesurgeon places the pedicle screw or any related surgical instruments forthe placement of the pedicle screw too close to the safe zone or withinthe safe zone, the area can be highlighted or another visual or acousticalert can be triggered by the software.

Alternatively, the software can place the pedicle screw automatically orsemiautomatically on the virtual display of the patient using criteriasuch as location of the pedicle screw including its tip in the vertebralbody, location of the pedicle screw including its tip in relationship toa spinal/vertebral body fracture, location of the pedicle screwincluding its tip in relationship to a superior endplate, location ofthe pedicle screw including its tip in relationship to an inferiorendplate, location of the pedicle screw including its tip inrelationship to the an anterior and/or posterior vertebral cortex,location of the pedicle screw including its tip in relationship to avessel, location of the pedicle screw including its tip in relationshipto the aorta, location of the pedicle screw including its tip inrelationship to the inferior vena cava, location of the pedicle screwincluding its tip in relationship to neural structures, the thecal sac,nerve roots and/or the spinal cord, distance, area or volume between thepedicle screw including its tip to a spinal/vertebral body fracture,distance, area or volume between the pedicle screw including its tip toa superior endplate, distance, area or volume between of the pediclescrew including its tip to an inferior endplate, distance, area orvolume between the pedicle screw including its tip to the an anteriorand/or posterior vertebral cortex, distance, area or volume between thepedicle screw including its tip to a vessel, distance, area or volumebetween the pedicle screw including its tip to the aorta, distance, areaor volume between the pedicle screw including its tip to the inferiorvena cava, distance, area or volume between the pedicle screw includingits tip to neural structures, the thecal sac, nerve roots and/or thespinal cord. The software can use the information on location ordistance or area or volume can also to select the size, width, diameteror length of the pedicle screw in the virtual surgical plan. Safe zonecriteria can be defined for the foregoing criteria, for example 1, 2 or3 or more mm from a cortex or a neural structure. If the software cannotplace the pedicle screw or any related surgical instruments for theplacement of the pedicle screw without violating one of the safe zonesor places it too close to the safe zone, the area can be highlighted oranother visual or acoustic alert can be triggered by the software. Thesurgeon can then manually adjust the virtual position of the pediclescrew or any related surgical instruments for the placement of thepedicle screw such as an awl, a probe, a needle, a wire, a tap and thelike.

The virtual surgical plan can only simulate the final desired placementof the pedicle screw(s) and any related rods. The desired trajectory ofany surgical instruments used for placing the pedicle screw such as anawl, a probe, a needle, a wire, a tap and the like can then be projectedduring the surgery based on the virtual surgical plan and the finaldesired placement position of the pedicle screw(s) and any related rods.

In some embodiments of the invention, each instrument or, for example,the principal instruments used for the placement of the pedicle screw(s)and/or the rods can be displayed during the surgery in the virtualdisplay. The physical instruments seen through the OHMD can be alignedwith the corresponding virtual instruments displayed by the OHMD,optionally in 3D, stereoscopic or non-stereoscopic, thereby achievingthe desired surgical alterations, for example according to the virtualsurgical plan.

FIGS. 17A-17D are illustrative flow charts of select options andapproaches for performing spine surgery in a mixed reality environment.In FIG. 17A, pre-operative patient visit, imaging, pre-operativeplanning 184, a surgeon evaluates a patient and sets the indication forspinal fusion using pedicle screws and spinal rods 185. Optionallyspinal radiographs 186 and/or 3D imaging, e.g. CT or MRI 187, can beobtained. Optionally the data can be segmented 188 and 191. Optionally2D data can be used 192. Bone contours can be derived automatically,semi-automatically or manually 189 from the radiographs 189 or CT or MRI193. Optionally, sensitive structures such as nerve roots and vesselscan be determined 194 and superimposed on the display of the 2D or 3Dbone data 198. Bone contours from radiographs and other imaging studiessuch as CT or MRI can optionally be cross-registered, e.g. usingcoordinate transfer or using registration in a common coordinate system190. Optionally, 2D projections of 3D data can be generated, for exampleto generate matching projections that can align with and/or besuperimposed with intra-operative radiographs 195. Optionally, a surgeonor operator can select points or landmarks or surfaces forintra-operative registration 196. Bone contours 189 and/or 193 and otherdata, e.g. 198, 197, 196 can be used to develop a virtual surgical planfor placement of the pedicle screw(s) and rod(s) 199. Optionally, theshape of one or more structures used for intra-operative registrationcan be derived 200 using software as described for example in DataSegmentation. Optionally, a patient specific template can be generatedfor the spine 201, as described, for example in WO9325157A1.

In FIG. 17B, intra-operative virtual surgical plan, imaging, landmarks,registration, cross-reference of virtual and live patient data 215, thedata from FIG. 17A, e.g. 189, 193, 194, 195, 199, 200, can be importedinto a workstation 202. The virtual data of the patient can also beimported, optionally including virtual instrument data, virtual devicedata and/or the virtual surgical plan 203. The OHMD can be connected tothe workstation 204 and can, optionally, display unregistered virtualdata 204. The patient can be positioned on the OR table, optionally inthe same position as that used for pre-operative imaging 205. Step 205can optionally be performed before 202, 203 and 204. Optionally, one ormore spinous processes or other bone landmarks or skin references can beidentified 206. Optionally, intra-operative imaging can be performed 207using, for example, x-rays or CT/O-arm imaging 207. Optionally, anincision can be performed over a spinous process and a patient specificmarker or template, an optical marker or other markers can be appliedfor registration 208 and 209. Landmarks, e.g. ones used in the virtualsurgical plan 199, can be identified 211, and can optionally becross-referenced or registered with landmarks identified byintra-operative imaging or patient specific markers or optical markersor other markers 210 and 212, for example in a common coordinate system,e.g. with the OHMD, or in different coordinate systems using coordinatetransfers. The patient can then be registered in a common, e.g. first,coordinate system 213. Optionally, markers can be attached to rigidstructures fixed to the spine and/or landmarks 231.

In FIG. 17C, continuation of intra-operative virtual surgical plan,imaging, landmarks, registration, cross-reference of virtual and livepatient data 215, after the registration of patient landmarks 213 one ormore OHMD/s can be registered in relationship to the patient or patientlandmarks 214, e.g. using spatial mapping or optical markers ornavigation markers or combinations thereof or any other registrationtechnique described in the application. Actual surgical instruments suchas awls and pins and implants such as pedicle screws and rods can alsobe registered 232. A 2D or 3D display can be generated, which caninclude hidden subsurface anatomy, e.g. of a vertebral body, pedicle,facet joints, virtual surgical instruments and virtual implants 216.These can be superimposed with and aligned with the corresponding livedata of the patient, e.g. the center of a pedicle in which an awl or ascrew can be placed in a predetermined position 216. Stereoscopic 217and non-stereoscopic 218 displays can be generated. Multiple viewers cansee the virtual data and the live data superimposed using multiple OHMDseach displaying the virtual data with the view perspective matching theview perspective of the live data for the individual viewer 216, 217,218. The viewer(s) can move their head freely and the OHMD worn by eachviewer can remain registered with the live data using, for example, oneor more of IMUs attached to the OHMD, room mapping, spatial mapping,e.g. of the surgical site or the patient or both, optical markers ornavigation markers 219. Instruments or implants, e.g. pedicle screws orrods, can also be tracked using, for example, IMUs, LEDs, opticalmarkers, or navigation markers 220. The display of the OHMD can beadjusted in real time, e.g. 30 frames per second or more, based on headmovement or instrument or device movement or combinations thereof 221.The surgeon can obtain a down the barrel view of a pedicle for placingtools, such as pins, or screws, for example in real time 222. A skinincision can be performed over select pedicle or multiple spine levels223.

In FIG. 17D, continuation of intra-operative virtual surgical plan,imaging, landmarks, registration, cross-reference of virtual and livepatient data 215, the surgeon can, for example, advance an awl towardsthe entry point for a pedicle screw 224. The actual or physical awl canbe aligned with a virtual awl 225. Other physical instruments can bealigned with their corresponding virtual instrument or, for example, anintended path or endpoint 226. Consecutive surgical steps can beexecuted aligning physical with virtual tools, instruments or implants227. Optionally, portions of the physical instrument that are hiddeninside or by the tissue can be displayed in the virtual display in theaugmented reality system using, for example, the alignment informationfrom the visible portions of the instrument 228. For this purpose,optical markers or navigation markers can, for example, be attached tothe instrument to register it and compute its hidden portions. Thephysical or actual pedicle screw can be placed aligned with orsuperimposed with the hidden subsurface anatomy, e.g. the pedicle, or avirtual pedicle screw, or an intended path or endpoint or combinationsthereof 229. The physical spinal rod can be placed aligned with orsuperimposed onto a virtual spinal rod 230; optionally, the spinal rodcan be placed aiming at virtual representations of the rod receptacle orreceiving or holding or attachment mechanisms of the pedicle screw(s).The rod receptacle or receiving or holding or attachment mechanisms canbe magnified by the OHMD for this purpose, for example around a centralaxis or central point, to facilitate aiming of the physical rod. Thehidden portions of the physical rod can be virtually displayed by theOHMD, optionally also magnified, and aimed at the rod receptacle orreceiving or holding or attachment mechanisms.

Any of the registration techniques and/or techniques described in theembodiments including implantable and attachable markers, calibrationand registration phantoms including optical markers, navigation markers,infrared markers, RF markers, LEDs with image capture and IMUs can beapplied for spinal surgery and procedures. For example, in a spinalsurgery or procedure, one or more patient specific markers or templatescan be applied to one or more spinous processes or articular processesor transverse processes or other spinal structures, for example througha small incision. By applying the patient specific markers or templatesto the corresponding structure(s) on the patient, reliableidentification of spinal levels is possible, optionally withoutintraoperative imaging. Moreover, pedicle screws and related instrumentsor vertebroplasty or kyphoplasty needles and trocars and relatedinstruments can be placed reliably following a trajectory or desiredposition of the pedicle screws and related instruments or vertebroplastyor kyphoplasty needles and trocars projected by the OHMD using anoptional virtual surgical plan. Of note, reliable identification ofspinal levels and reliable placement of pedicle screws, rods, andrelated instruments and or vertebroplasty or kyphoplasty needles andtrocars is also possible using the OHMD with the other registration andcross-referencing techniques described in the invention or known in theart.

The same steps and OHMD guided spinal procedures are also possible usingthe OHMD with the other registration and cross-referencing techniquesdescribed in the invention or known in the art, such as, for example,registration using anatomic landmarks or registration or calibrationphantoms including optical markers or image capture, optionally usingoptical markers, or surgical navigation.

In some embodiments of the invention, the registration of virtualpatient data and live patient data using the techniques described hereincan be repeated after one or more surgical steps have been performed inan OHMD guided spinal procedure. In this case, the surgically alteredtissue or tissue surface or tissue contour or tissue perimeter or tissuevolume or other tissue features in the live patient can be matched to,superimposed onto and/or registered with the surgically altered tissueor tissue surface or tissue contour or tissue perimeter or tissue volumeor other tissue features in the virtual data of the patient, e.g. in avirtual surgical plan developed for the patient. The matching,superimposing and/or registering of the live data of the patient and thedigital holograms of the patient's tissue and/or surgical site includinghidden and/or obscured parts after the surgical tissue alteration can beperformed using the same techniques described in the foregoing or any ofthe other registration techniques described in the specification or anyother registration technique known in the art.

Hip Replacement

Any of the registration techniques and/or techniques described in theembodiments can be applied for hip replacement surgery, includingresurfacing, partial and total hip replacement including implantable andattachable markers, calibration and registration phantoms includingoptical markers, navigation markers, infrared markers, RF markers,patient specific markers, LEDs with image capture and IMUs. For example,one or more patient specific markers or templates or optical markers canbe applied to the edge of the acetabulum, the inside of the acetabulumor a pelvic wall. Similarly, one or more patient specific markers ortemplates or optical markers can be applied to a greater trochanter, alesser trochanter, a femoral shaft or a femoral neck. By applying theone or more patient specific markers or templates and/or optical markersto the corresponding structures on the patient, virtual data and livedata can be effectively cross-referenced and/or registered in a commoncoordinate system, for example with one or more OHMDs. By registeringthe patient specific marker or template and/or optical marker inrelationship to the OHMD also or by using any of the other registrationtechniques or techniques described herein or known in the art, the OHMDcan display or superimpose the desired position, location, orientation,alignment and/or trajectory of any surgical instrument used during hipreplacement. For example, an acetabular reamer can be applied at apredetermined angle, with the long axis of the reamer typically matchingthe desired acetabular cup angle, offset, medial or lateral positionand/or anteversion and/or inclination, e.g. from a virtual surgical planfor the patient.

FIGS. 18A-18F are illustrative examples of displaying a virtualacetabular reaming axis using one or more OHMDs and aligning a physicalacetabular reamer with the virtual reaming axis for placing anacetabular cup with a predetermined cup angle, offset, medial or lateralposition and/or anteversion and/or inclination. FIG. 18A shows a firstsurgeon's view, e.g. through an OHMD, onto the patient's exposedacetabulum 280. Note also the anterior superior iliac spine 281 and thesymphysis pubis 282, which can optionally be used for registrationpurposes, for example using attached optical markers or navigationmarkers. In FIG. 18B, the first surgeon can see a virtual acetabularreaming axis 283 through the OHMD, which can be oriented in apredetermined manner to achieve a predetermined acetabular cup angle,offset, medial or lateral position and/or anteversion and/orinclination, e.g. from a virtual surgical plan for the patient. In FIG.18C, the first surgeon aligns the physical acetabular reamer shaft 284so that its central axis is aligned or superimposed with the virtualacetabular reaming axis thereby placing the reamer head 285 in theacetabulum in a predetermined position and orientation for apredetermined acetabular cup angle, offset, medial or lateral positionand/or anteversion and/or inclination.

FIG. 18D shows a second surgeon's view with his or her respective viewperspective of live data and virtual data through the OHMD onto thepatient's exposed acetabulum 280. Note also the anterior superior iliacspine 281 and the symphysis pubis 282, which can optionally be used forregistration purposes, for example using attached optical markers ornavigation markers. In FIG. 18E, the second surgeon can see the virtualacetabular reaming axis 283 through the OHMD, which can be oriented in apredetermined manner to achieve a predetermined acetabular cup angle,offset, medial or lateral position and/or anteversion and/orinclination, e.g. from a virtual surgical plan for the patient. Thevirtual acetabular reaming axis is projected with a view angle or viewperspective matching the view angle or view perspective of the live dataof the patient seen by the second surgeon. In FIG. 18F, the secondsurgeon can see how the physical acetabular reamer shaft 284 is alignedby the first surgeon so that its central axis is aligned or superimposedwith the virtual acetabular reaming axis thereby placing the reamer head285 in the acetabulum in a predetermined position and orientation for apredetermined acetabular cup angle, offset, medial or lateral positionand/or anteversion and/or inclination.

Thus, the surgeon can hold the physical acetabular reamer seeing thelive data through the OHMD; at the same time, the OHMD can display orproject a digital hologram of the corresponding virtual acetabularreamer with the virtual acetabular reamer aligned and oriented toachieve a desired acetabular cup position, e.g. anteversion,inclination, as optionally defined in a virtual surgical plan.Alternatively, the OHMD can display a partial (e.g. broken or dotted) orcomplete 2D or 3D outline of the virtual acetabular reamer or one ormore placement indicators, e.g. lines indicating the predeterminedplacement position and orientation of the acetabular reamer, e.g. avirtual predetermined medial border or placement or position, a virtualpredetermined lateral border or placement or position, a virtualpredetermined anterior border or placement or position, a virtualpredetermined posterior border or placement or position, a virtualpredetermined superior border or placement or position and/or a virtualpredetermined inferior border or placement or position and/or or virtualpredetermined rim position and/or a virtual predetermined central axisorientation or position and/or a virtual predetermined anteversion.

The surgeon can now align the physical acetabular reamer with thevirtual acetabular reamer or its 2D or 3D outline or placement indicatoror predetermined or virtual reaming axis displayed by the OHMD so thatthe physical acetabular reamer is substantially superimposed or alignedwith or oriented along the virtual acetabular reamer or its 2D or 3Doutline or placement indicator or virtual reaming axis. The OHMD canalso indicate the desired reaming depth as optionally defined in avirtual surgical plan. The desired reaming depth can be displayed by theOHMD, e.g. as a virtual red border to which the physical reamer can beadvanced. If the reaming surface of the physical reamer is not visiblesince it is hidden by tissue, e.g. soft-tissue or bone, it can beestimated based on the visible portions of the physical reamer and itcan be optionally displayed by the OHMD, e.g. using a different colorthan the display of the virtual reamer or the virtual “red border” forthe reaming depth. The physical reaming depth of the physical reamer canalso be measured, for example via image capture or mechanical datacapture of a numeric scale on the physical reamer which indicatesreaming depth, or by attaching IMUs or one or more optical markers, RFtags or retro-reflective markers for navigation to the reamer and bycomparing physical measured reaming depth to the virtual surgical plan.The OHMD can indicate when the desired reaming depth has been achieved,for example with a visual or acoustic signal. One or more opticalmarkers can also be attached to the shaft of the acetabular reamer. Bymeasuring the position of the one or more optical markers, e.g. twooptical markers in two different locations along the shaft of thereamer, the long axis of the physical acetabular reamer can bedetermined using image or video capture and can be compared to thepredetermined virtual reaming axis to achieve a desired cup placement,including a desired offset and/or cup angle and/or anteversion.

The physical acetabular cup can be placed by obtaining substantial ornear substantial superimposition with the virtual acetabular cup or its2D or 3D outline or placement indicator(s) projected by the OHMD using,for example, the virtual surgical plan for the patient, whereby thevirtual acetabular cup or its 2D or 3D outline or placement indicator(s)show the desired anteversion and inclination. Depending on the surgicalapproach, e.g. anterior, posterior or posterolateral, only thoseportions of the virtual acetabular cup can be displayed that correspondto the portions of the physical acetabular cup which would be visiblefor the surgical approach or surgical site. Optionally, the physicalvalues of anteversion and inclination can be numerically displayed, e.g.by the OHMD, showing, for example, the desired values for the patientfrom the virtual surgical plan and the physical values based on thephysical cup or trial cup position, location, orientation, and/oralignment. If there is a visual discrepancy, i.e. incompletesuperimposition between virtual cup displayed by the OHMD and thephysical or trial cup, or a numeric discrepancy, e.g. in virtual cupanteversion and/or inclination from the virtual surgical plan versusphysical cup anteversion and/or inclination, the surgeon can correct theposition, location, orientation, and/or alignment of the physical cupprior to impaction.

By utilizing the 3D anatomic information of the patient from thepre-operative data or by using intra-operative measurements, for exampleoptical markers for determining a center of rotation of a hip joint orfor determining a desired anteversion, the surgeon can work moreaccurately in this manner, thereby reducing, for example, the need foroffset or asymmetric liners.

Of note, the same steps and OHMD guided acetabular procedures are alsopossible using the OHMD with any of the registration andcross-referencing techniques described in the invention and known in theart, such as, for example, registration using anatomic landmarks orregistration or calibration phantoms including optical markers or imagecapture, optionally using optical markers, or surgical navigation orpatient specific markers or intra-operative imaging.

Any of the registration techniques or techniques described hereinincluding implantable and attachable markers, calibration andregistration phantoms including optical markers, navigation markers,infrared markers, RF markers, patient specific markers, LEDs with imagecapture and IMUs can be applied for registering the patient's proximalfemur in relationship to, for example, one or more OHMDs worn by thesurgeon and/or is assistants, and/or in relationship to one or moresurgical instruments, pins, drills, saws, reamers, impactors, broachesand the like and/or in relationship to one or more femoral or acetabularimplants, including metal and/or polyethylene components. For example,by applying one or more optical markers and/or patient specific markersor templates to a greater trochanter, a lesser trochanter, a femoralshaft or a femoral neck, virtual and physical live patient data can becross-referenced on the femoral side. Optionally, a pin or a screw canbe inserted into the proximal femur, e.g. in a greater trochanter, whichcan be used as a reference for registration, for example if an opticalmarker or patient specific marker moves. Optical markers can beoptionally attached to the pin or screw. Multiple pins or screws can beused in this manner. The virtual surgical plan can include a desiredneck cut location for a particular femoral component. The neck cut canbe designed or selected to avoid any offset issues and to maintain thepatient's leg length in the virtual surgical plan. By registering theoptical marker and/or patient specific marker or template inrelationship to the OHMD also, e.g. in a common coordinate system withthe OHMD, the surgical site, the proximal femur, the OHMD can display orsuperimpose and/or project digital holograms showing the desired orpredetermined position, location, orientation, alignment and/ortrajectory or predetermined plane of any surgical instrument including asaw for performing the femoral neck cut. After successful registrationof virtual and live data of the patient using any of the techniques ortechniques described herein, the OHMD can show the desired 3D trajectoryincluding the desired location, entry point and angles in x, y and zdirection for the femoral neck cut or the OHMD can display one or moredigital holograms of a virtual cut plane and/or a virtual saw or sawblade in the position, location, angular orientation, and trajectory(e.g. as a dotted line or arrow) defined in the surgical plan which thesurgeon can then match with the physical saw, i.e. the surgeon canorient and align the physical saw so that it will be aligned with orsubstantially superimposed onto the virtual saw (see also FIGS. 4A-4C).Alternatively, the OHMD can show a digital hologram of a partial (e.g.broken or dotted) or complete 2D or 3D outline of the virtual saw orplacement indicators, e.g. lines indicating the predetermined placementposition and orientation of the saw, e.g. a virtual predetermined medialplacement or position, a virtual predetermined lateral placement orposition, a virtual predetermined anterior placement or position, avirtual predetermined posterior placement or position, a virtualpredetermined superior placement or position and/or a virtualpredetermined inferior placement or position. Alternatively, the OHMDcan show a digital hologram of a virtual femoral neck cut plane.

Optionally, for example once the entry point on the femoral neck hasbeen defined or the desired location, orientation and/or direction ofthe saw has been determined with assistance from the OHMD, the surgeoncan apply a standard saw guide to the femoral neck to facilitate theneck cut. Alternatively, the OHMD can display a digital hologram of avirtual femoral neck saw guide or its corresponding 2D or 3D outline orplacement indicators in its desired position or location on the femoralneck. The physical saw guide can then be aligned with the correspondingvirtual saw guide or its corresponding 2D or 3D outline or placementindicators placed in the desired position, orientation and angulationbased on the virtual surgical plan of the patient. The virtual saw guidecan have the same or similar shape and/or one or more dimensions orplanes as the physical saw guide. Once the physical saw guide issubstantially superimposed in position with the virtual saw guide or itscorresponding 2D or 3D outline or placement indicators displayed by theOHMD, the surgeon can optionally pin the physical saw guide in place andperform the neck cut. By executing the neck cut using one of theseapproaches which utilize accurate 3D anatomic information of the patientfrom the pre-operative scan and/or intra-operative measurementsincluding registration, e.g. using optical markers, leg length andoffset can be more accurately preserved or addressed.

Similarly, the OHMD can project the desired position, location,orientation and trajectory of any virtual femoral reamers and impactors.Alternatively, the OHMD can only show a partial (e.g. broken or dotted)or complete 2D or 3D outline of the virtual femoral reamers or impactorsor placement indicators, e.g. lines indicating the predeterminedplacement position and orientation of the reamers or impactors, e.g. avirtual predetermined medial placement or position, a virtualpredetermined lateral placement or position, a virtual predeterminedanterior placement or position, a virtual predetermined posteriorplacement or position, a virtual predetermined superior placement orposition or a virtual predetermined inferior placement or position or avirtual reaming axis, e.g. a central axis through the reamer shaft. TheOHMD can also display a digital hologram of a predetermined virtualreaming and/or broaching axis, which can provide a desired femoralcomponent position including one or more of an offset and/or anteversionincluding, for example, composite anteversion for both femoral andacetabular components. The virtual femoral reamers and impactors canhave the same or similar shape and dimensions as the physical femoralreamers and impactors. The surgeon can then match the position,location, orientation and trajectory (e.g. indicated by a dotted line oran arrow in the virtual data) of the physical femoral reamers andimpactors with the virtual reamers and impactors or their corresponding2D or 3D outlines or placement indicators or a virtual reaming orbroaching axis, thereby reducing the possibility of mal-seating of thefemoral stem and possibly incorrect femoral anteversion, incorrectfemoral offset or femoral component angulation or leg lengthdiscrepancy. In some embodiments of the invention, the surgeon can alignthe OHMD so that the view angle is perpendicular to the femoral shaftaxis or, alternatively, the femoral neck axis. The OHMD can then displaya bulls-eye or target like structure whereby the surgeon will aim thefemoral reamers, impactors, femoral trials and the physical femoralcomponent to be located in the center of the bulls-eye or target. TheOHMD can display the desired entry point, e.g. with regard to medial orlateral, anterior or posterior location on the cut femoral neck, and/orentry angle based on the virtual surgical plan including, for example,the virtual femoral component placement. The OHMD can also display thedesired femoral version, for example via a solid or dotted line orarrows on the cut femoral neck surface or in relationship to the cutfemoral neck surface. The desired femoral version can also be displayedby the OHMD by displaying one or more digital holograms of the femoralreamers, impactors, femoral trials and the final femoral component ortheir respective 2D or 3D outlines or placement indicators in thedesired virtual location and orientation including femoral version basedon the virtual surgical plan. In this manner, the surgeon can align thephysical femoral reamers, physical impactors, physical femoral trialsand the physical final femoral component to be substantially aligned orsuperimposed with the digital holograms of the one or more virtualfemoral reamers, virtual impactors, virtual femoral trials and virtualfinal femoral component thereby achieving a result near the desiredfemoral version and, optionally, leg length based on the virtualsurgical plan.

All of the foregoing steps and OHMD guided femoral procedures are alsopossible using the OHMD with any of the other registration andcross-referencing techniques described in the invention or known in theart, such as, for example, registration using anatomic landmarks orimplantable and attachable markers, calibration and registrationphantoms including optical markers, navigation markers, infraredmarkers, RF markers, patient specific markers, LEDs with image captureand IMUs.

In some embodiments of the invention, an ultrasound scan can be used toderive the shape information used for designing and producing thepatient specific template, e.g. for use on the acetabular side or thefemoral side. Optionally, the ultrasound scan can be obtained in supineand/or upright position. By obtaining the ultrasound scan in uprightposition, information about femoro-acetabular alignment and orientationcan be obtained under weight-bearing positions including, for example,femoral or acetabular anteversion, femoral/acetabular/hip flexion,extension, abduction, adduction and/or rotation. By obtaining theultrasound scan in supine position, information about femoro-acetabularalignment and orientation can be obtained under non-weight-bearingpositions including, for example, femoral or acetabular anteversion,femoral/acetabular/hip flexion, extension, abduction, adduction and/orrotation. By comparing data from one or more upright and one or moresupine ultrasound scans, e.g. by comparing the relative movement ofcorresponding anatomic landmarks, information can be obtained aboutpelvic tilt. The information from the upright and/or supine scan can beused for selecting the desired femoral and acetabular componentsincluding, for example, the shape and length of the femoral neck, theoffsets, the femoral head component, as well as the shape of theacetabular component, including, for example, offset, mesialized,lateralized, or rimmed acetabular components. The information from theupright and/or supine scan can be used for developing or adjusting thevirtual surgical plan, for example by changing the predetermined cupposition based on the upright scan information or based on informationon pelvic tilt. Similar information can be obtained using supine andupright x-rays studies.

Optionally, the information from the upright and/or supine imaging datacan be used to assess information on pelvic tilt, which in turn can beintroduced into the surgical plan and component selection in order toavoid or minimize the risk of postoperative complications such ascomponent dislocation.

Thus, by performing hip replacement using the different embodiments ofthe invention, it is possible for the surgeon to conduct the surgerywith high accuracy thereby reducing the possibility of commoncomplications in hip replacement such as offset error or wrongacetabular or femoral anteversion leading to hip dislocation or leglength discrepancy. Optionally, the OHMD can also display sensitivevascular or neural structures, thereby reducing the possibility ofvascular injury or, for example, sciatic nerve injury.

In some embodiments of the invention, the registration of virtualpatient data and live patient data using the techniques described hereincan be repeated after one or more surgical steps have been performed inan OHMD guided hip replacement procedure. In this case, the surgicallyaltered tissue or tissue surface or tissue contour or tissue perimeteror tissue volume or other tissue features in the live patient can bematched to, superimposed onto and/or registered with the surgicallyaltered tissue or tissue surface or tissue contour or tissue perimeteror tissue volume or other tissue features in the virtual data of thepatient, e.g. in a virtual surgical plan developed for the patient. Thematching, superimposing and/or registering of the live data of thepatient and the virtual data of the patient after the surgical tissuealteration can be performed using the same techniques described in theforegoing or any of the other registration techniques described in thespecification or any other registration technique known in the art. Forexample, the re-registration can be performed using a cut bone surface,e.g. a cut femoral neck using the surface shape, surface area orperimeter or other feature, optionally measured with image capture ormechanical or physical probes, to match, superimpose and/or register thelive patient data and the virtual patient data prior to performingsubsequent surgical steps, e.g. a reaming, milling or impacting of thefemoral canal for placement of a femoral component. For example, there-registration can be performed using a milled bone surface, e.g. amilled acetabulum using the surface shape, surface area or perimeter orother feature, optionally measured with image capture or mechanical orphysical probes, to match, superimpose and/or register the live patientdata and the virtual patient data prior to performing subsequentsurgical steps, e.g. a placement of an acetabular component includingtrial components.

Knee Replacement, Partial or Total

With knee replacement general alignment and orientation recommendationsexist, some of which have been summarized in a review (Gromov et al.Acta Orthop 2014, 85, 5, 480-487): Neutral overall coronal alignment iscurrently the gold standard, and a neutral mechanical axis of the leg or2-7° valgus anatomical tibial femoral axis can be targeted. The femoralcomponent can be placed in 2-8° coronal valgus with respect to thefemoral anatomic axis (e.g., 2°, 3°, 4°, 5°, 6°, 7°, 8°, 2-3°, 2-4°,2-5°, 2-6°, 2-7°, 2-8°, 3-4°, 3-5°, 3-6°, 3-7°, 3-8°, 4-5°, 5-6°, 5- 7°,5-8°, 6-7°, 6-8°, 7-8°) and >3 mm of implant component overhang over thebone should be avoided. The tibial component can be placed in neutralcoronal alignment (90°) with maximum bone coverage and minimal, if any,implant component overhang. In the sagittal plane, the femoral componentcan be placed with 0-3° of flexion (e.g. 0°, 1°, 2°, 3°, 0-1°, 0-2°,0-3°, 1-2°, 1-3°, 2-3°), and the tibial slope can be 0-7° (e.g. 0°, 1°,2°, 3°, 4°, 5°, 6°, 7°, 0-1°, 0-2°, 0-3°, 0-4°, 0-5°, 0-6°, 0-7°, 1-2°,1-3°, 1-4°, 1-5°, 1-6°, 1-7°, 2-3°, 2-4°, 2-5°, 2-6°, 2-7°, 3-4°, 3-5°,3-6°, 3-7°, 4-5°, 4-6°, 4-7°, 5-6°, 5-7°, 6-7°). Internal rotation ofthe femoral component should be avoided, as the femoral component shouldbe placed in 2-5° of external rotation in relation to surgicaltransepicondylar axis (e.g., 2°, 3°, 4°, 5°, 2-3°, 2-4°, 2-5°, 3-4°,3-5°, 4-5°). Excessive tibial rotation with respect to neutraltransverse axis of the tibia, tibial tubercle axis and also combinedinternal tibiofemoral rotation should also be avoided.

Any of the registration techniques and/or techniques described in theembodiments can be applied for knee replacement, e.g. resurfacing,partial and total knee replacement procedures, including implantable andattachable markers, calibration and registration phantoms includingoptical markers, navigation markers, infrared markers, RF markers,patient specific markers, LEDs with image capture and IMUs. For example,one or more optical marker and/or patient specific markers or templatescan be applied to the distal femur, for example the distal anteriorcortex and/or the superior trochlea, optionally along with anyosteophytes when present. Similarly, one or more optical markers and/orpatient specific markers or templates can be applied to the proximaltibia, e.g. the anterior tibial cortex, for example in the tibialplateau area, optionally along with any osteophytes when present, or atibial spine. By applying the one or more optical markers and/or patientspecific markers or templates or any of the other registrationtechniques including implantable and attachable markers, calibration andregistration phantoms, navigation markers, infrared markers, RF markers,LEDs with image capture and IMUs to the corresponding structures on thepatient, virtual data, e.g. derived from pre-operative imaging, and livedata can be effectively cross-referenced for knee replacement surgeryand can be, for example registered in a common coordinate system, e.g.with one or more OHMDs worn by the surgeon and his or her surgicalassistants and nurses. By registering optical marker and/or the patientspecific marker or template in relationship to the OHMD also, the OHMDcan display or superimpose the desired position, location, orientation,alignment and/or axes and/or trajectory of any surgical instrument usedduring knee replacement.

In some embodiments of the invention, an ultrasound scan can be used toobtain the shape information of the distal femur and/or the proximaltibia and/or the patella, for example for designing, selecting ormanufacturing a patient specific marker or template. For example, ahandheld ultrasound or an ultrasound probe attached to a holding device,stand, tripod or the like can be used to image the distal anteriorcortex and the superior trochlea of the femur, optionally along with anyosteophytes when present. The ultrasound device can then be used tooptionally image the proximal tibia, e.g. the anterior tibial cortex,for example in the tibial plateau area, optionally along with anyosteophytes when present. The ultrasound device can then be used tooptionally image the patella, e.g. the patellar surface, the wholepatella or portions of the patella, for example the superior pole orinferior pole, medial or lateral edge, optionally along with anyosteophytes when present. The ultrasound data can optionally besegmented. For example, bone shape and/or cartilage shape as well as,optionally, meniscal shape, when present, can be derived. Moreover,information about ligament location and/or morphometry, including, butnot limited to, the origin, insertion, location, length, movement withflexion, extension, rotation of the knee, of the medial collateralligament, lateral collateral ligament, anterior cruciate ligament,posterior cruciate ligament, patellofemoral ligament or tendon andquadriceps insertion can optionally also be captured with ultrasound.

In some embodiments, the shape information derived from the ultrasounddata can optionally be used to design, select and/or manufacture apatient specific marker or template, for example one that fits on thedistal anterior cortex and the superior trochlea of the femur of thepatient, optionally along with any osteophytes when present; or one thatfits on the proximal tibia of the patient, e.g. the anterior tibialcortex, for example in the tibial plateau area, optionally along withany osteophytes when present; or one or more that fits on the patella ofthe patient, e.g. the patellar surface, the whole patella or portions ofthe patella, for example the superior pole or inferior pole, medial orlateral edge, optionally along with any osteophytes when present.

Optionally, the ultrasound probe can also be used to image portions ofthe patient's hip joint, for example, to identify the center of the hipjoint. Optionally, the ultrasound probe can also be used to imageportions of the patient's ankle joint, for example to identify the anklemortise or the center of the ankle joint or the ⅓ or ⅔ equidistantdistance of the ankle joint in the coronal plane or select radii ordistance from the medial or lateral ankle mortise.

Optionally, the ultrasound scan(s) of the knee, optionally the hip andoptionally the ankle can be obtained in supine or in upright position.By obtaining the ultrasound scan or scans in upright position,optionally more accurate information on mechanical axis alignment, inparticular during weight-bearing, can be obtained. For example, varus orvalgus deformity of the knee can be more pronounced under weight-bearingconditions. Correction of varus or valgus deformity using mechanicalaxis information under weight-bearing conditions can be more accuratethan correction of varus or valgus deformity based on non-weight-bearinginformation. This information can be beneficial when planning anydesired mechanical axis corrections.

Optionally, the location of the ultrasound probe can be captured whileperforming the hip scan and/or the ankle scan and, optionally, the kneescan, for example using optical markers with image capture or videocapture, retro-reflective markers, infrared markers or RF markers orother tracking means used in conjunction with a surgical navigationsystem or, for example, using image capture, e.g. integrated into,attached to, coupled to or separate from an OHMD, or using one or moreIMUs. By imaging the hip joint and the ankle joint, and, optionally, theknee joint in this manner and by capturing information of the ultrasoundprobe location and orientation using one or more attached markers duringthe ultrasound scan, it is possible to derive information on themechanical axis and/or the anatomic axis of the patient's leg and kneejoint.

In some embodiments, information from an ultrasound, e.g. of the distalfemur, proximal tibia, and/or patella, can be combined or fused withinformation from another imaging modality, e.g. an MRI, CT or x-ray.X-rays can include x-rays in prone, supine, non-weight-bearing positionor in standing, weight-bearing position. X-rays can be limited to theknee only. X-rays can be obtained in different poses of the knee, e.g.in extension and at different flexion angles, weight-bearing ornon-weight-bearing. Flexion/extension x-rays can, for example, be usedto derive information about the rotational axes of the knee, e.g. anepicondylar or trochlear axis. X-rays can also include other portions ofthe lower extremity or the entire lower extremity, such as a standingfull length x-ray of the leg in weight-bearing position. A standing fulllength x-ray of the leg in weight-bearing position can be used toidentify the center of the hip joint as well as the ankle mortise, forexample to estimate or derive a mechanical axis and/or an anatomic axisof the knee. In some embodiments of the invention, mechanical axisand/or anatomic axis and/or rotational axis information of the kneeobtained from x-rays can be included in a patient specific marker ortemplate derived from ultrasound. For example, a patient specific,ultrasound derived surface of the patient-specific marker can fit to aselect anatomic region of the patient, e.g. a distal femur includingportions of the superior trochlea or an anterior tibial cortex, forexample in the tibial plateau area. One or more external facing surfacesof the patient specific marker or template can have a standard shape andcan, optionally, include markers or indicators to show an anatomic axisof the knee of the patient, a mechanical axis of the knee of thepatient, a desired new mechanical axis of the knee of the patient afterthe surgery is performed, e.g. as defined in an optional virtualsurgical plan, and/or a rotational axis of the knee of the patientand/or a desired new rotational axis of the knee of the patient afterthe surgery is performed, e.g. as defined in an optional virtualsurgical plan. These external markers or indicators including opticalmarkers can then optionally be used during the surgery to confirm, forexample, a desired mechanical axis correction or rotational axiscorrection or combinations thereof. An image and/or video capture systemattached to, integrated with, coupled to or separate from an OHMD canoptionally be used to identify such corrections using, for example, oneor more of the optical markers or indicators on the patient specificmarker or template and, optionally to compare them to a virtual surgicalplan. Any deviations or differences from the virtual surgical plan canbe identified and the surgeon or operator can optionally performmodifications to the surgical technique, e.g. using additional ligamentreleases, bone cuts or different implant components including, forexample, different medial, lateral or combined insert heights, insertshapes, spacers, and augments.

In some embodiments of the invention, the accuracy of the placement ofan optical marker or a patient specific marker can be checked during thesurgery. For example, in a knee replacement, the optical marker orpatient specific marker can be placed on a distal femur or a proximaltibial or combinations thereof. A visual or an optical marker, e.g. anLED or a laser light, can indicate a mechanical axis of the patient,e.g. by projecting an arrow or a beam towards the center of the hipand/or the ankle. Alternatively, a mechanical marker, e.g. a femoralalignment rod pointing towards the hip or a tibial alignment rodpointing towards the ankle, can be used to indicate the mechanical axisof the patient as determined using the optical marker or patientspecific marker. The femoral and/or tibial alignment rod can beintegral, attachable or physically or visually linkable to the opticalmarker or patient specific marker. One or more optical markers can beintegrated into or attached to a femoral and/or tibial alignment rod.

An intraoperative x-ray or an intra-operative ultrasound or anintra-operative CT can then be used to determine the physical center ofthe hip and/or the physical center of the ankle in the live patient onthe OR table and, optionally, the patient's physical mechanical axisprior to any corrections. If the projected mechanical axis from opticalmarker or the patient specific marker coincides with the physical centerof the hip and/or the physical center of the ankle, the placement or theinformation from the optical marker or patient specific marker isaccurate. If the projected mechanical axis from the optical markerand/or patient specific marker does not coincide with the physicalcenter of the hip and/or the physical center of the ankle, the placementof the optical marker and/or patient specific marker is not accuratelyplaced and can be repositioned. The degree or amount of differencebetween the physical and the projected center of the hip and/or theankle can be used to determine the amount of correction of placementneeded. Alternatively, the optical marker and/or patient specific markercan remain in place; however, a correction can be applied to anysubsequent registration, wherein the correction is based on the degreeor amount of difference between the physical (from the intraoperativeimaging study) and the projected center of the hip and/or the ankle(from the optical marker(s) and/or patient specific marker(s)). Someoneskilled in the art can recognize that these types of corrections inplacement or corrections can be applied to other measurements, e.g.rotational axes, and other joints.

Once any correction of placement inaccuracies of the optical markersand/or patient specific markers has been performed, if applicable, theintended axis correction, e.g. a correction of the patient's abnormalmechanical or rotational axis or both, can be executed on.

Femur

In some embodiments of the invention, once the femur is registered usingany of the techniques described in the invention and/or any of the otherregistration techniques described in the invention or known in the art,including implantable and attachable markers, calibration andregistration phantoms including optical markers, navigation markers,infrared markers, RF markers, patient specific markers, LEDs with imagecapture and IMUs, the OHMD can display a virtual distal femoral cutblock for performing the distal femoral cut.

FIGS. 19A-19D provide an illustrative, non-limiting example of the useof virtual surgical guides such as a distal femoral cut block displayedby an OHMD and physical surgical guides such as physical distal femoralcut blocks. FIG. 19A shows live data of a patient with a distal femur300 exposed during knee replacement surgery, a medial condyle 301, alateral condyle 302 and a trochlea 303. In FIG. 19B, one or more OHMDscan display a virtual distal femoral cut block, e.g. in a stereoscopicmanner for the left eye and the right eye of the surgeon(s) creating aform of electronic hologram of the virtual surgical guide, i.e. thevirtual distal cut block. The virtual distal femoral cut block 304 inthis example is an outline of the physical distal femoral cut block withsubstantially similar dimensions as those of the physical distal femoralcut block. The virtual distal femoral cut block 304 is aligned based atleast in part on coordinates of a predetermined position for guiding thedistal femoral cut, for example for achieving a predetermined varus orvalgus correction and/or a predetermined femoral component flexionrelative to the distal femur and, for example, its anatomic orbiomechanical axes. In FIG. 19C, the physical surgical guide 305, i.e.the physical distal femoral cut block 305 (solid line) in this example,can be moved and aligned to be substantially superimposed with oraligned with the virtual surgical guide 304, i.e. the virtual distalfemoral cut block 304 (broken line) in this example. The hidden areas ofthe knee joint 306, obscured or hidden by the physical distal femoralcut block 305, can optionally also be displayed by the OHMD. In FIG. 19D, the physical distal femoral cut block 305 can be attached to thedistal femoral bone using two pins 307. These pins 307 can be used forsubsequent surgical steps, for example for referencing a flexion gap oran extension gap or for ligament balancing. The OHMD can stop displaythe virtual surgical guide, i.e. the virtual distal femoral cut block inthis example, but can optionally continue display the hidden anatomy306.

The virtual distal femoral cut block can have the same or similar shapeand one or more dimensions and one or more planes as the physical distalfemoral cut block. Alternatively, the OHMD can only show a partial (e.g.broken or dotted) or complete 2D or 3D outline of the virtual distalfemoral cut block or placement indicators, e.g. lines or planesindicating the predetermined placement position and orientation of thedistal femoral cut block, e.g. a virtual predetermined medial border orplacement or position, a virtual predetermined lateral border orplacement or position, a virtual predetermined anterior border orplacement or position, a virtual predetermined posterior border orplacement or position, a virtual predetermined superior border orplacement or position and/or a virtual predetermined inferior border orplacement or position. In the virtual surgical plan, the distal femoralcut will typically be perpendicular to the mechanical axis of the femurin order to restore mechanical axis alignment, unless the surgeondesires to preserve a mild varus deformity, for example, as can be thecase with partial or some total knee replacements, or unless the surgeonuses a different alignment approach, e.g. kinematic alignment, or unlessthe surgeon desires to maintain a certain amount of pre-existing varusor valgus alignment in a patient. The surgeon can then take the physicaldistal femoral cut block and substantially align or superimpose thephysical distal femoral cut block with the virtual distal femoral cutblock or its 2D or 3D outline or its placement indicators displayed bythe OHMD. Once adequate alignment or superimposition of the physicaldistal femoral cut block with the virtual distal femoral cut block orits 2D or 3D outline or its placement indicators displayed by the OHMDbased on the patient's virtual surgical plan is achieved, the surgeoncan pin or attach the physical distal femoral cut block to the bone andperform the cut. By utilizing preoperative 3D data information orintra-operative measurements of combinations of both for the alignmentof the physical distal femoral cut block with the assistance of theOHMD, the surgeon can perform the distal femoral cut in an accuratemanner, without the need for intramedullary rods or patient specificinstrumentation for performing the cut. Alternatively, the OHMD candisplay a digital hologram of a virtual cut plane corresponding to thedistal femoral cut and the surgeon can align the saw blade with thedigital hologram of the virtual distal femoral cut plane.

Optionally, the OHMD can display a digital hologram of a virtual femoralalignment rod, which can extend from the distal femur to the hip joint.The surgeon can compare the alignment of the virtual femoral alignmentrod with the physical femoral alignment rod in the live patient andassess if both align with the center of the hip joint of the livepatient. If the virtual and the physical femoral alignment rod are notaligned with each other and/or the center of the hip joint, the surgeoncan check the accuracy of alignment of the physical alignment rod in thelive patient, the accuracy of registration of live data of the patientand virtual data of the patient and/or the accuracy of the virtualsurgical plan. The surgeon can then optionally make adjustments to thealignment of the physical alignment rod in the live patient, theregistration or the virtual surgical plan.

The surgeon can then, for example, select to display or project adigital hologram of the virtual femoral AP cut block in the OHMD. Thevirtual femoral AP cut block can have the same or similar shape anddimensions as the physical femoral AP cut block. The OHMD can displaythe virtual femoral AP cut block or a partial (e.g. broken or dotted) orcomplete 2D or 3D outline of the virtual distal femoral cut block orplacement indicators, e.g. planes or lines indicating the predeterminedplacement position and orientation of the AP femoral cut block, e.g. avirtual predetermined medial border or placement or position, a virtualpredetermined lateral border or placement or position, a virtualpredetermined anterior border or placement or position, a virtualpredetermined posterior border or placement or position, a virtualpredetermined superior border or placement or position and/or a virtualpredetermined inferior border or placement or position. The virtualsurgical plan can include the predetermined position and rotation forthe virtual femoral AP cut block. The rotation of the femoral AP cutblock can determine the rotation of the resultant anterior and posteriorfemoral cuts in relationship to the femoral axis and, with that, candetermine the femoral component implant rotation. The OHMD can displaythe virtual femoral AP cut block or its 2D or 3D outline or one or moreplacement indicators.

FIGS. 20A-20C provide an illustrative, non-limiting example of the useof virtual surgical guides such as an AP femoral cut block displayed byan OHMD and physical surgical guides such as physical AP cut blocks forknee replacement. FIG. 20A shows live data of a patient with a distalfemur 300 exposed during knee replacement surgery after a distal femoralcut creating a planar distal surface 310, a medial condyle 301, alateral condyle 302 and a trochlea 303. In FIG. 20B, one or more OHMDscan display a virtual femoral AP cut block 312, e.g. in a stereoscopicmanner for the left eye and the right eye of the surgeon(s) creating aform of electronic or digital hologram of the virtual surgical guide,i.e. the virtual femoral AP cut block 312. The virtual femoral AP cutblock 312 in this example is an outline of the physical femoral AP cutblock with similar dimensions, edges, or planes as those of the physicalfemoral AP cut block. The virtual femoral AP cut block 312 is alignedbased at least in part on coordinates of a predetermined position forguiding the different bone cuts, e.g. an anterior cut, posterior cutand/or chamfer cuts depending on the configuration of the physicalfemoral AP cut block, for example for achieving a predetermined femoralcomponent rotation. In FIG. 20C, the physical surgical guide 314, i.e.the physical femoral AP cut block 314 (solid line) in this example, canbe moved and aligned to be substantially superimposed with or alignedwith the virtual surgical guide 312, i.e. the virtual femoral AP cutblock 312 (broken line) in this example. The physical femoral AP cutblock can be attached to the distal femoral bone using pins (not shown)and the cuts can be performed. Subsequent surgical steps can optionallybe referenced based on one or more of the cuts executed using thephysical femoral AP cut block.

The surgeon can align or substantially superimpose the physical femoralAP cut block with the digital hologram of the virtual femoral AP cutblock or its 2D or 3D outline or one or more placement indicatorsprojected by the OHMD. Once adequate alignment or superimposition of thephysical AP cut block with the virtual AP cut block or its 2D or 3Doutline or one or more placement indicators displayed by the OHMD hasbeen achieved, the surgeon can pin the physical AP cut block and performthe cuts. By utilizing preoperative 3D data information orintra-operative information, e.g. from optical marker and image or videocapture measurements, for the position, alignment and rotation of thephysical femoral AP cut block with the assistance of the OHMD, thesurgeon can perform the anterior and posterior femoral cuts in a highlyaccurate manner, thereby achieving accurate rotational alignment of thefemoral component. The same approaches and display options, e.g. virtualcut blocks, 2D or 3D outline or one or more placement indicators, can beapplied to all subsequent femoral preparation steps including chamfercuts and chamfer cut blocks.

Of note, similar steps and OHMD guided femoral procedures are alsopossible using the OHMD with any of the other registration andcross-referencing techniques described in the invention or known in theart, for example intraoperative image guidance.

Tibia

In some embodiments of the invention, once the tibia is registered usingany of the techniques described in the invention or known in the art,including, for example, implantable and attachable markers, calibrationand registration phantoms including optical markers, navigation markers,infrared markers, RF markers, patient specific markers, LEDs with imagecapture and IMUs, the OHMD can display a virtual proximal tibial cutblock for performing the proximal tibial cut. Alternatively, the OHMDcan only show a partial (e.g. broken or dotted) or complete 2D or 3Doutline of the virtual proximal tibial cut block or placementindicators, e.g. planes or lines indicating the predetermined placementposition and orientation of the proximal tibial cut block, e.g. avirtual predetermined medial border or placement or position, a virtualpredetermined lateral border or placement or position, a virtualpredetermined anterior border or placement or position, a virtualpredetermined posterior border or placement or position, a virtualpredetermined superior border or placement or position and/or a virtualpredetermined inferior border or placement or position. The virtualproximal tibial cut block can have the same or similar shape anddimensions as the physical proximal tibial cut block or it can have atleast one or more dimensions or planes that are identical to thephysical proximal tibial cut block or guide.

FIGS. 21A-21F provide an illustrative, non-limiting example of the useof virtual surgical guides such as a virtual proximal tibial cut guidedisplayed by an OHMD and physical surgical guides such as physicalproximal tibial cut guide. FIG. 21A shows live data of a patient with aproximal tibia 330 exposed during knee replacement surgery, a medialtibial plateau 331, a lateral tibial plateau 332 and a medial tibialspine 333 and a lateral tibial spine 334. In FIG. 21B, one or more OHMDscan display a virtual proximal tibial cut guide, e.g. in a stereoscopicmanner for the left eye and the right eye of the surgeon(s), creating aform of electronic hologram of the virtual surgical guide, i.e. thevirtual proximal tibial cut guide. The virtual proximal tibial cut guide336 in this example can be an outline of the physical proximal tibialcut guide with substantially similar dimensions as those of the physicalproximal tibial cut guide. The virtual proximal tibial cut guide 336 isaligned based at least in part on coordinates of a predeterminedposition for guiding the proximal tibial cut, for example for achievinga predetermined varus or valgus correction and/or a predetermined sloperelative to the proximal tibia and, for example, its anatomic orbiomechanical axes. In FIG. 21C, the physical surgical guide 338, i.e.the physical proximal tibial cut guide 338 (solid line) in this example,can be moved and aligned to be substantially superimposed with oraligned with the virtual surgical guide 336, i.e. the virtual proximaltibial cut guide 336 (broken line) in this example. Note two pin holes339 in the physical proximal tibial cut guide 338. In FIG. 21D, thephysical proximal tibial cut guide 338 can be attached to the proximaltibia bone using two pins 340. These pins 307 can be used for subsequentsurgical steps, for example for referencing a flexion gap or anextension gap or for ligament balancing. In FIG. 21E, an alternativeembodiment is shown to FIG. 21B. One or more OHMDs can display a virtualproximal tibial cut plane 342, e.g. in a stereoscopic manner for theleft eye and the right eye of the surgeon(s), creating a form ofelectronic hologram of the virtual tibial cut plane. The virtualproximal tibial cut plane 342 in this example is parallel with andsubstantially aligned and superimposed with the predetermined cut planefor the physical proximal tibial cut guide. The virtual proximal tibialcut plane 342 is aligned based at least in part on coordinates of apredetermined position for guiding the proximal tibial cut, for examplefor achieving a predetermined varus or valgus correction and/or apredetermined slope relative to the proximal tibia and, for example, itsanatomic or biomechanical axes. A physical saw blade or a slot foraligning the physical saw blade in a physical proximal tibial cut guideor an open guide area for accommodating the saw blade in a physicalproximal tibial cut guide can then be aligned and at least partiallysuperimposed with the virtual proximal tibial cut plane 342. In FIG.21F, an alternative embodiment is shown to FIG. 21B. One or more OHMDscan display two or more virtual drills or pins 344 for placement in theproximal tibia, e.g. in a stereoscopic manner for the left eye and theright eye of the surgeon(s), creating a form of electronic hologram ofthe virtual tibial pins or drills. The virtual drills or pins 344 inthis example can be an outline or a projected path of the physical pinsor drills that can be used to fixate a physical proximal tibial cutguide to the proximal tibia. The virtual drills or pins 344 are alignedbased at least in part on coordinates of a predetermined position forguiding the proximal tibial cut, for example for achieving apredetermined varus or valgus correction and/or a predetermined sloperelative to the proximal tibia and, for example, its anatomic orbiomechanical axes. The physical drills or pins (not shown) can then bealigned and superimposed with the virtual drills or pins 344 and placedin the proximal tibia. A physical proximal tibial cut guide can then beattached to the physical pins and the proximal tibial cut can beexecuted.

In some embodiments of the invention, a physical and a correspondingvirtual proximal tibial guide or a physical and a corresponding virtualdistal femoral guide can also be pin guides, wherein the physical guidecan be used to place two or more pins in the bone for attaching physicalcut guides for subsequent surgical steps. The embodiments for aligningphysical with virtual guides, as shown for example in FIGS. 19B and 19C,20B and 20C, and 21B and 21C, can also be applied to pin guides.

Someone skilled in the art can recognize that the use of virtual andphysical surgical guides, including cut guides and pin guides, can beapplied to any joint of the human body and the spine.

In the virtual surgical plan, the proximal tibial cut can beperpendicular to the mechanical axis of the tibia in order to restoreneutral mechanical axis alignment, unless the surgeon desires topreserve a mild varus deformity, for example, as can be the case withpartial or some total knee replacements, or unless the surgeon uses adifferent alignment approach, e.g. kinematic alignment, or unless thesurgeon desires to maintain a certain amount of pre-existing varus orvalgus alignment in a patient. The surgeon can then take the physicalproximal tibial cut block and substantially align or superimpose thephysical proximal tibial cut block with the virtual proximal tibial cutblock or its 2D or 3D outline or its placement indicators displayed bythe OHMD. The virtual surgical plan and/or the intraoperativemeasurements can optionally determine not only the alignment of theproximal tibial cut in relationship to the mechanical axis of the leg,but can also determine the anterior-posterior slope with which theproximal tibia is cut in sagittal direction. In some embodiments, thesurgeon, the operator or semi-automatic or automatic software may electto cut the proximal tibia with a fixed sagittal slope, e.g. 5 degrees or7 degrees or 3 degrees, for example with a Cruciate Retaining (CR) kneereplacement system. Or the surgeon, the operator or semi-automatic orautomatic software may elect to cut the proximal tibia with a fixedsagittal slope, e.g. 0 degrees or 2 degrees or 3 degrees, for examplewith a Posterior Substituting (PS) knee replacement system. Or thesurgeon, the operator or semi-automatic or automatic software may electto cut the proximal tibia with a patient specific slopes, which can beidentical to or derived from the medial slope of the native, un-operatedmedial tibial plateau, the lateral slope of the native, un-operatedlateral tibial plateau, or combinations or averages thereof. Onceadequate alignment or superimposition of the physical proximal tibialcut block with the virtual representation of the virtual proximal tibialcut block or its 2D or 3D outline or its placement indicators displayedby the OHMD based on the patient's virtual surgical plan and/orintra-operative measurements is achieved, the surgeon can pin thephysical proximal tibial cut block and perform the cut, which can thenreflect an alignment with the desired mechanical axis correction and thedesired tibial slope. By utilizing preoperative 3D data informationand/or intraoperative measurements and/or information for the alignmentof the physical proximal tibial cut block with the assistance of theOHMD, the surgeon can perform the proximal tibial cut in an accuratemanner, without the need for intramedullary rods or patient specificinstrumentation for performing the cut. At the same time, the surgeonretains the ability to perform intraoperative adjustments, which can beas simple as manually moving the distal or other femoral cut blocks ormoving the proximal tibial cut block or other tibial cut blocks, forexample also with use of a stylus like device, e.g. for checking andmeasuring slope. Any such adjustment can be checked against the virtualsurgical plan and/or the intraoperative measurements, by displaying inthe OHMD, for example, the final desired implant position or thepredetermined position of the corresponding virtual surgical instrumentsfor which the adjustment is contemplated in the physical surgicalinstrument. Any difference in alignment between any virtual surgicalinstrument and any physical surgical instrument can be indicated innumeric values by the OHMD, e.g. distance in millimeters or angles indegrees, e.g. difference in external rotation of the femoral component.Any subsequent steps in the virtual surgical plan can be modified in theevent the surgeon or operator elected to perform an adjustment, e.g. oftibial slope or femoral or tibial resection levels.

Of note, the same steps and OHMD guided tibial procedures are alsopossible using the OHMD with any of the other registration andcross-referencing techniques described in the invention or known in theart, for example using intraoperative image guidance and implantable andattachable markers, calibration and registration phantoms includingoptical markers, navigation markers, infrared markers, RF markers,patient specific markers, LEDs with image capture and IMUs.

A tibial template or tibial base trial can be used to prepare theproximal tibia for accepting the tibial implant component. A drill canbe used to remove the bone in the center of the proximal tibia to acceptthe central bore of the keel of the tibial component. A keel punch canbe used to punch out the space to accept the keel wings of the tibialcomponent. The final seating and orientation of the tibial keel and keelwings can determine tibial implant rotation. Accurate tibial rotation,for example aligned with the rotation axis of the native knee, is animportant objective for avoiding postoperative pain.

In some embodiments of the invention, the OHMD can display a digitalhologram of a virtual tibial template or virtual tibial base trial aswell as virtual tibial drill towers and virtual keel punches. Othervirtual tibial preparation instruments can be displayed depending on theconfiguration and surgical technique of the knee replacement systemused. Alternatively, the OHMD can only show a partial (e.g. broken ordotted) or complete 2D or 3D outline of the virtual tibial template orvirtual tibial base trial as well as virtual tibial drill towers andvirtual keel punches or other virtual tibial preparation instruments orplacement indicators, e.g. planes or lines indicating the predeterminedplacement position and orientation of the tibial template or tibial basetrial as well as tibial drill towers and keel punches or other tibialpreparation instruments, e.g. a virtual predetermined medial border orplacement or position, a virtual predetermined lateral border orplacement or position, a virtual predetermined anterior border orplacement or position, a virtual predetermined posterior border orplacement or position, a virtual predetermined superior border orplacement or position and/or a virtual predetermined inferior border orplacement or position. The virtual tibial template or tibial base trialas well as virtual tibial drill towers and virtual keel punches andother virtual tibial preparation instruments can have the same orsimilar shape and dimensions as the physical tibial template or physicaltibial base trial as well as physical tibial drill towers and physicalkeel punches and physical tibial preparation instruments. In the virtualsurgical plan, the virtual tibial template or tibial base trial as wellas virtual tibial drill towers and virtual keel punches and virtualtibial preparation instruments can be aligned in a manner to achieveclose to zero tibial rotation error of the final, physical tibial trayimplanted in relationship to the native rotation axis of the tibia ofthe un-operated knee, if intended. The surgeon or operator has theoption to deviate from zero rotation and can add optionally 1, 2, 3 ormore degrees of internal or external tibial component rotation to thevirtual surgical plan and/or the intra-operative measurements.

For each step of the tibial preparation, the OHMD can display digitalholograms of the virtual tibial instrument(s) used or its (their) 2D or3D outline or its (their) placement indicators along with its (their)desired alignment and rotation based on the virtual surgical plan. Thesurgeon can then align or superimpose the corresponding physical tibialinstrument with the virtual tibial instrument(s) or its (their) 2D or 3Doutline or its (their) placement indicators thereby achieving thedesired alignment and/or rotation of the physical tibial instrument inrelationship to the virtual surgical plan and/or the intraoperativemeasurements. All virtual tibial preparation tools and instrumentsincluding virtual tibial templates or virtual tibial base trials as wellas virtual tibial drills, drill towers or saws and keel punches can bedisplayed using digital holograms by the OHMD if desired. Alternatively,the OHMD can display digital holograms of a 3D contour or placementindicators of the virtual tibial instruments. Optionally, the OHMD canonly display the key instruments used for setting tibial componentalignment and rotation. By utilizing preoperative 3D data informationand/or intra-operative measurements and/or information for the position,alignment and rotation of the virtual tibial preparation instruments,the tibial trials and final tibial components or their respective 2D or3D outlines or placement indicators displayed with the assistance of theOHMD, the surgeon can perform the physical tibial preparation in anaccurate manner by matching physical instruments and components with thealignment and rotation of the virtual instruments and components ortheir respective 2D or 3D outlines or placement indicators, therebyachieving accurate rotational alignment of the tibial component.

Optionally, the OHMD can display a digital hologram of a virtual tibialalignment rod, which can extend from the proximal tibia to the anklejoint. The surgeon can compare the alignment of the virtual tibialalignment rod with the physical tibial alignment rod in the live patientand assess if both align with the desired location in the ankle joint ofthe live patient. If the virtual and the physical tibial alignment rodare not aligned with each other and/or the desired location in the anklejoint, the surgeon can check the accuracy of alignment of the physicalalignment rod in the live patient, the accuracy of registration of livedata of the patient and virtual data of the patient and/or the accuracyof the virtual surgical plan and/or the intra-operative measurements.The surgeon can then optionally make adjustments to the alignment of thephysical alignment rod in the live patient, the registration or thevirtual surgical plan.

Of note, the same steps and OHMD guided tibial procedures are alsopossible using the OHMD with the other registration andcross-referencing techniques described in the invention or known in theart including implantable and attachable markers, calibration andregistration phantoms including optical markers, navigation markers,infrared markers, RF markers, patient specific markers, LEDs with imagecapture and IMUs.

Patella

In some embodiments of the invention, one or more optical markers and/orpatient specific markers or templates or combinations thereof can beapplied to the patella or patellar surface or portions of the patella,for example the superior pole or inferior pole, medial or lateral edge,optionally along with any osteophytes when present. By applying the oneor more optical markers and/or patient specific markers or templates tothe corresponding structures on the patient or using any of the othertechniques and techniques for registration described in the invention orknown in the art, e.g. implantable and attachable markers, calibrationand registration phantoms, navigation markers, infrared markers, RFmarkers, LEDs with image capture and IMUs, virtual data and live datacan be effectively cross-referenced for patellar replacement or partialor complete resurfacing. By registering the optical marker and/orpatient specific marker or template in relationship to the OHMD, e.g. ina common coordinate system with the OHMD and the femur, tibia andpatella, or by registering the OHMD in relationship to the live data andvirtual data of the patient using any of the registration techniquesdescribed in the invention, the OHMD can display or superimpose digitalholograms indicating the desired position, location, orientation,alignment and/or trajectory of any surgical instrument used duringpatellar replacement or partial or complete resurfacing, including witha virtual display of the patellar preparation instrument, a 2D or 3Doutline of the patellar preparation instrument or a virtual display ofpredetermined placement indicators.

In some embodiments of the invention, once the patella is registeredusing any of the techniques described in the invention, including, forexample, implantable and attachable markers, calibration andregistration phantoms including optical markers, navigation markers,infrared markers, RF markers, patient specific markers, LEDs with imagecapture and IMUs, the OHMD can display or project a digital hologram ofa virtual patellar clamp, patellar tool, patellar cutting device,patellar milling device, predetermined milling axis and/or patellar cutblock or other patellar preparation instrument for performing thepatellar cut or patellar preparation. Alternatively, the OHMD can onlyshow a partial (e.g. broken or dotted) or complete 2D or 3D outline ofthe virtual patellar clamp, patellar tool, patellar cutting device,patellar milling device, and/or patellar cut block or other patellarpreparation instrument or placement indicators, e.g. lines indicatingthe predetermined placement position and orientation of the virtualpatellar clamp, patellar tool, patellar cutting device, patellar millingdevice, and/or patellar cut block or other patellar preparationinstrument, e.g. a virtual predetermined medial border or placement orposition, a virtual predetermined lateral border or placement orposition, a virtual predetermined anterior border or placement orposition, a virtual predetermined posterior border or placement orposition, a virtual predetermined superior border or placement orposition and/or a virtual predetermined inferior border or placement orposition. The digital holograms of the virtual patellar clamp, patellartool, patellar cutting device, patellar milling device, and/or patellarcut and/or other patellar preparation instrument block can have the sameor similar shape and at least one or more dimensions or planes that areidentical to those of the corresponding physical patellar clamp,patellar tool, patellar cutting device, patellar milling device, and/orpatellar cut block and/or other patellar preparation instrument. In thevirtual surgical plan, the patellar cut or milling can be planned, forexample at a desired resection depth or angle selected for a particularpatellar implant or replacement and/or a particular patient anatomy,and/or based on patellar shape, patellar tracking, patellofemoralkinematics or knee rotation axes. The surgeon can then take the physicalpatellar clamp, patellar tool, patellar cutting device, patellar millingdevice, and/or patellar cut block and/or other patellar preparationinstrument and substantially align or superimpose the physical patellarclamp, patellar tool, patellar cutting device, patellar milling device,and/or patellar cut block and/or other patellar preparation instrumentwith the corresponding virtual patellar clamp, patellar tool, patellarcutting device, patellar milling device, and/or patellar cut blockand/or other patellar preparation instrument or its respective virtualcontour or placement indicators displayed by the OHMD. Once adequatealignment or superimposition of the physical patellar clamp, patellartool, patellar cutting device, patellar milling device, and/or patellarcut block and/or other patellar preparation instrument with the digitalholograms of the virtual patellar clamp, patellar tool, patellar cuttingdevice, patellar milling device, and/or patellar cut block and/or otherpatellar preparation instrument or its respective contour or placementindicators displayed by the OHMD based on the patient's virtual surgicalplan and/or intra-operative measurements is achieved, the surgeon canoptionally pin or fixate the physical virtual patellar clamp, patellartool, patellar cutting device, patellar milling device, and/or patellarcut block and/or other patellar preparation instrument and perform thecut or milling. By utilizing preoperative 3D data information orintraoperative data and/or measurements or combinations thereof for thealignment of the physical virtual patellar clamp, patellar tool,patellar cutting device, patellar milling device, and/or patellar cutblock and/or other patellar preparation instrument with the assistanceof the OHMD, the surgeon can perform the patellar cut or milling in ahighly accurate manner.

The patellar procedures described in the invention can also beimplemented using any of the other registration techniques described inthe invention or known in the art including implantable and attachablemarkers, calibration and registration phantoms including opticalmarkers, navigation markers, infrared markers, RF markers, patientspecific markers, LEDs with image capture and IMUs. For example, usingan image and/or video capture system integrated into, attached to,coupled to or separate from the OHMD, it is possible to image thepatellar shape or surface or contour. The information can be compared topre-operative imaging information about patellar shape or surface orcontour and a match can optionally be performed for purposes ofregistration. Any of the other registration techniques described in theinvention or known in the art including, but not limited to, surgicalnavigation can be used.

Optionally, an IMU including, for example, pyrometers, magnetometers andaccelerometers can be applied to the patella during the surgery orpre-operatively.

In some embodiments of the invention, the registration of virtualpatient data and live patient data using the techniques described hereincan be repeated after one or more surgical steps have been performed inan OHMD guided knee replacement procedure. In this case, the surgicallyaltered tissue or tissue surface or tissue contour or tissue perimeteror tissue volume or other tissue features in the live patient can bematched to, superimposed onto and/or registered with the surgicallyaltered tissue or tissue surface or tissue contour or tissue perimeteror tissue volume or other tissue features in the virtual data of thepatient, e.g. in a virtual surgical plan developed for the patient. Thematching, superimposing and/or registering of the live data of thepatient and the virtual data of the patient after the surgical tissuealteration can be performed using the same techniques described in theforegoing or any of the other registration techniques described in thespecification or any other registration technique known in the artincluding implantable and attachable markers, calibration andregistration phantoms including optical markers, navigation markers,infrared markers, RF markers, patient specific markers, LEDs with imagecapture and IMUs. For example, the re-registration can be performedusing a cut bone surface, e.g. a cut distal femur using the surfaceshape, surface area or perimeter or other feature to match, superimposeand/or register the live patient data and the virtual patient data priorto performing subsequent surgical steps.

Someone skilled in the art that the same concepts and embodimentsdescribed for spinal surgery, knee replacement and hip replacement canbe applied to other surgeries of the human body, e.g. repair orreconstruction of the anterior cruciate ligament, posterior cruciateligament, other ligaments, shoulder replacement, ankle replacement,and/or wrist replacement. For example, an OHMD can display or projectdigital holograms of one or more surgical instruments, trial implants orimplant components or one or more outlines or axes of the surgicalinstruments, trial implants, or implant components or digital hologramsof predetermined start point, predetermined start position,predetermined start orientation/alignment, predetermined intermediatepoint(s), predetermined intermediate position(s), predeterminedintermediate orientation/alignment, predetermined end point,predetermined end position, predetermined end orientation/alignment,predetermined path, predetermined plane, predetermined cut plane,projected contour/outline/cross-section/surfacefeatures/shape/projection, predetermined depth marker or depth gauge,predetermined angle/orientation/rotation marker, predetermined axis,e.g. rotation axis, flexion axis, extension axis, predetermined axis ofthe virtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, estimated/projected non-visualizedportions for one or more devices/implants/implant components/surgicalinstruments/surgical tools, and/or one or more of a predetermined tissuechange/alteration for a shoulder replacement, wherein the one or moredigital holograms can be used to determine a humeral resection, armlength, glenoid component version, orientation and/or position, humeralcomponent version, orientation and/or position.

An OHMD can display or project digital holograms of one or more surgicalinstruments, trial implants or implant components or one or moreoutlines or axes of the surgical instruments, trial implants, or implantcomponents or digital holograms of a predetermined start point,predetermined start position, predetermined start orientation/alignment,predetermined intermediate point(s), predetermined intermediateposition(s), predetermined intermediate orientation/alignment,predetermined end point, predetermined end position, predetermined endorientation/alignment, predetermined path, predetermined plane,predetermined cut plane, projected contour/outline/cross-section/surfacefeatures/shape/projection, predetermined depth marker or depth gauge,predetermined angle/orientation/rotation marker, predetermined axis,e.g. rotation axis, flexion axis, extension axis, predetermined axis ofthe virtual surgical tool, virtual surgical instrument including virtualsurgical guide or cut block, virtual trial implant, virtual implantcomponent, implant or device, estimated/projected non-visualizedportions for one or more devices/implants/implant components/surgicalinstruments/surgical tools, and/or one or more of a predetermined tissuechange/alteration for an ankle replacement, wherein the one or moredigital holograms can be used to display or project an predeterminedtibial resection and/or talar resection with desired coordinates,angles, orientation and/or alignment to achieve a desired anklealignment including at least one of a coronal plane implant componentalignment, sagittal plane implant component alignment including flexionand axial plane component alignment or rotation

Repair and/or Reconstruction of the Anterior Cruciate Ligament

The following embodiments and description on performing an ACL repairusing aspects of the invention are only meant to be exemplary and arenot meant to be limiting of the invention. Any of the methods describedin the specification can be applied or used. Any of the imagingtechniques, patient positioning techniques, registration techniques,methods for developing surgical plans including at different flexion andextension or rotation angles, displaying virtual and live patient datacan be applied to any of the other embodiments in this specification,including, for example, knee replacement, hip replacement, pedicle screwplacement and spinal fusion, and vertebroplasty or kyphoplasty.

Tears of the anterior cruciate ligament (ACL) represent one of the mostcommon injuries of the human knee. They can result in knee instability,for example with flexing or bending the knee. Surgical treatment of anACL tear can include the placement of an autograft or an allograft oranother graft material. ACL repair can be performed using the so-calledsingle bundle technique or double bundle technique.

The objective of ACL repair or reconstruction is the restoration ofnormal knee kinematics in patients with unstable or ACL deficient knees.Anatomical reconstruction of the ACL may help restore normal kneekinematics and reduce the possibility of developing osteoarthritis ofthe knee after ACL injury. Anatomically, two different portions orbundles of the ACL have been described, an antero-medial bundle and apostero-lateral bundle.

ACL reconstruction can be performed using a so-called single bundletechnique or a double bundle technique. One of the objectives of thesurgical ACL reconstruction includes placing the graft tissue in anisometric position to restore knee function and to reduce thepossibility of postoperative graft complications and graft failure.Placement of the graft near or at the location of the native, torn ACLhas the benefit that the ACL graft is placed in a location that ensureprimarily isometric ligament function which can help the long-termsurvival of the graft. The surgeon will typically try to place the ACLgraft in the location and/or orientation of the native, torn ACL. Afemoral and a tibial bone tunnel need to be placed in order toaccommodate the graft. The femoral tunnel extends from the posteriorfemoral cortex into the area of the femoral notch, typically where theorigin of the native ACL was located. The tibial canal extends typicallyfrom the medial tibial spine, the attachment of the native, torn ACL, tothe anteromedial tibial cortex. An anchor can typically be placed in thearea where the graft enters the femoral bone and/or exits the tibialbone.

Tunnel positions can be chosen in a predetermined position and/ororientation to achieve such an isometric function. Tunnel positions canbe placed in a predetermined position and/or orientation so that thefemoral tunnel will exit the distal femur near the origin of the ACL.The tibial tunnel can be placed in a predetermined position and/ororientation so that it enters the proximal tibia near the insertion ofthe ACL on the medial tibial spine. The angle and/or orientation of thefemoral and/or tibial tunnel can be placed in a predetermined positionand/or orientation so that it is similar to the natural angle and/ororientation of the native ACL or, optionally, different from the naturalorientation of the native ACL of the patient. If a single bundletechnique is used, the angle and/or orientation of the femoral and/ortibial tunnel including their entry and exit areas can be directed inpredetermined positions and/or orientations so that the location and/ororientation of the graft is a compromise between the location and/ororientation of the antero-medial bundle of the ACL and thepostero-lateral bundle of the ACL. A trans-tibial technique can be usedas a method for tunnel placement, wherein the femoral tunnel can bedrilled in a predetermined position and/or orientation through thetibial tunnel. This has the benefit that both tunnels can be linked.Alternatively, the tibial tunnel can be drilled first in a predeterminedposition and/or orientation, for example through a small incision in theskin of the anterior tibia, followed by drilling of the femoral tunnel,for example through a small incision and portal into the knee joint.Optionally, the tunnel location can be placed in a predeterminedposition and/or orientation with arthroscopic visualization, for exampleby evaluating the location of residual ACL fibers on the femur and/or onthe tibia. Placement of the graft outside the intended location and/ororientation can be caused by incorrect placement of the femoral and/ortibial tunnel. Incorrect placement of one or both tunnels and incorrectplacement of the graft can lead to limitations in knee function andearly wear and tear of the graft.

FIGS. 22A AP and 22B lateral views demonstrating exemplary normal ACL360 including antero-medial and postero-lateral fibers. Curved brokenline on femoral side indicates intercondylar notch area/roof 361.

FIGS. 22C AP and 22D lateral views demonstrating exemplary ACL tunnels362 (solid straight lines) on femoral side and tibial side. Curvedbroken line on femoral side indicates intercondylar notch area/roof 361.

Imaging

In some embodiments of the invention, the patient can undergo apre-operative or intra-operative scan, e.g. a CT scan, an MRI scan or anultrasound scan. Optionally, the femoral and tibial bones can besegmented and displayed in two or three dimensions. In some embodiments,the origin and the insertion of the native, torn ACL can be identified.Alternatively or in addition, one or more portions of the torn nativeACL can be identified. The information can be used to develop a virtualsurgical plan for placement of the femoral and/or the tibial tunnel orthe graft using OHMD guidance, e.g. by displaying one or more virtualfemoral or tibial tunnels or one or more virtual grafts.

For example, if an MRI scan is used, the MRI data can be imported into asoftware program to segment the femoral and/or tibial bones. For thispurpose, a T1-weighted MRI sequence can be chosen without fatsuppression. On the T1-weighted sequence without fat suppression, themarrow space can display intermediate to high signal intensity. Themarrow space is bounded by low signal intensity cortical bone. The highintensity marrow space can be segmented, for example using athresholding algorithm or a seed growing algorithm or an active contouror level set technique or any other algorithm or technique known in theart. A two or three millimeter or other thickness cortical bone andsubchondral bone envelope can be added. The thickness envelope can beapplied using a reference database, e.g. for bones of known size ordimensions. The cortical bone or subchondral bone envelope can vary inthickness depending on the location on the tibia or the femur. Thethickness can be derived based on anatomic reference data.Alternatively, the cortical bone and subchondral bone can be segmentedusing any method and/or algorithm known in the art. Optionally, a 3Ddisplay of the data can be generated. Alternatively, the original 2Ddata can be displayed. The surgeon can use a pointer or marking tool tomark the original of the torn ACL and the insertion of the torn ACL. Thelocation of the origin and insertion of the antero-medial bundle and thepostero-lateral can be marked separately. Any ACL remnants or portionsthereof can be marked by the surgeon or operator.

If a CT scan is used, the CT data can be imported into a softwareprogram to segment the femoral and/or tibial bones using, for instance athresholding or isosurface algorithm. Optionally, an algorithm can beapplied that detects surface roughness and based on this informationidentified the femoral original of the ACL. Alternatively, the femoralsurface in the posterolateral femoral notch can be visually inspected onthe 2D or 3D images to identify the origin of the ACL. The medial tibialspine can be identified to mark the insertion of the ACL.

If an ultrasound is used, the femoral and tibial bones can be visualizedin 2D. The ultrasound data can optionally be imported into a softwareprogram to segment the femoral and/or tibial bones. The residual femoralfibers of the ACL can optionally be identified to determine the locationof the native ACL origin. Or the femoral surface roughness in thelocation of the ACL origin can be used for this purpose. The medialtibial spine can be identified to mark the insertion of the ACL. Anyother imaging test known in the art can be used.

Optionally, the medial and lateral femoral condyles can be identified onthe ultrasound images or ultrasound data; optionally, the medial andlateral tibial plateau can be identified on the ultrasound images orultrasound data. Other anatomic landmarks, surfaces and features (forexample as provided in the Table below entitled “Exemplary anatomiclandmarks, surfaces and features in the knee for registration of virtualand live data including, optionally, pre-operative and intraoperativeimaging data, for ACL Repair/Reconstruction” can be identified.Optionally, one or more of these anatomic landmarks, surfaces andfeatures of the distal femur and/or the proximal tibial can be used toidentify a standard femoral shape or a standard tibial shape bycomparing the one or more anatomic landmarks, surfaces and features withdata in a reference database of reference patients and/or referencefemoral shapes and/or reference tibial shapes and by selecting a 3Dmodel of the distal femur and/or proximal tibial that most closelymatches the selected anatomic landmarks, surfaces or features. In thismanner, the 3D shape of the patient's bones, e.g. the distal femurand/or the distal tibia, can be estimated without the need acquire 3Ddata or without the need of segmentation of the 3D data or limiting theamount of segmentation needed. The reference database can be, forexample, an anatomic reference database from cadaver data. The referencedatabase can also be, for example, scan data, e.g. acquired in the NIHOsteoarthritis Initiative or acquired from imaging data to generatepatient specific instruments for knee replacement.

If one or more x-rays are used, they can, for example, be obtained in anAP projection of the knee (or PA), and a lateral projection of the knee.Other views are possible, as known in the art, e.g. a tunnel view,Merchant view, patellar view, oblique views, standing views, supineviews, prone views. Optionally, the medial and lateral femoral condylescan be identified on the AP/PA and/or lateral and/or oblique views;optionally, the medial and lateral tibial plateau can be identified onthe AP/PA and/or lateral and/or oblique views. Other anatomic landmarks,surfaces and features (for example as provided in Table 12 can beidentified. Optionally, one or more of these anatomic landmarks,surfaces and features of the distal femur and/or the proximal tibial canbe used to identify a standard femoral shape or a standard tibial shapeby comparing the one or more anatomic landmarks, surfaces and featureswith data in a reference database of reference patients and/or referencefemoral shapes and/or reference tibial shapes and by selecting a 3Dmodel of the distal femur and/or proximal tibial that most closelymatches the selected anatomic landmarks, surfaces or features. In thismanner, the 3D shape of the patient's bones, e.g. the distal femurand/or the distal tibia, can be estimated without the need acquire 3Ddata or without the need of segmentation of the 3D data or limiting theamount of segmentation needed. The reference database can be, forexample, an anatomic reference database from cadaver data. The referencedatabase can also be, for example, scan data, e.g. acquired in the NIHOsteoarthritis Initiative or acquired from imaging data to generatepatient specific instruments for knee replacement.

Of note, the use 2D imaging data or 3D imaging data, e.g. x-ray,ultrasound, CT or MRI, in combination with one or more referencedatabases of 3D shape(s) of select anatomic structures, such as a bone,a cartilage, an organ for reducing or limiting or obviating the need foracquiring 3D data or for segmenting 2D or 3D data is applicable to anyembodiment of the invention throughout the specification including forall other clinical applications, e.g. hip replacement, knee replacement,spinal surgery, spinal fusion, vertebroplasty, kyphoplasty, fracturefixation, brain surgery, liver surgery, cancer surgery etc.

Virtual Surgical Plans

With the location of the origin and the insertion or the remnants of thepatient's native ACL identified using any of the foregoing methods orany other method known in the art, the surgeon or the software candevelop/generate a virtual surgical plan using the 2D or 3D imaging dataor, optionally, kinematic data, e.g. data simulating knee flexion and/orextension and/or rotation. For example, software can display the virtualdata, e.g. imaging data, of the patient. The surgeon or the software canoptionally select a desired size or diameter femoral tunnel and/ortibial tunnel for a given patient. The diameter and size of the tunnelcan be chosen, for example, based on the size of the patient's bone, thesize of the patient's tendon, e.g. if a tendon autograft iscontemplated, the size of the patient's patellar tendon, e.g. if apatellar autograft is contemplated, the size of the patient'ssemitendinosus tendon, e.g. if a semitendinosus autograft iscontemplated, or the expected size of an allograft or an artificialgraft or the expected biomechanical loads or stresses applied to thegraft; the same or similar or other parameters can also be used inchoosing a femoral and/or a tibial anchor for the graft, which caninclude one or more interference screws or other types of anchorsincluding button type anchors. The surgeon or the software canoptionally select a predetermined femoral or tibial tunnel locationand/or orientation, for example using the femoral origin of the native,torn ACL as an entry point in the femur and the medial tibial spine asan entry point into the tibia. Note, the term entry and exit point canbe used interchangeably in the specification.

The surgeon or the software can optionally select a desired size andlength graft, e.g. an allograft or an autograft, for a given patient.The diameter and size of the graft can be chosen, for example, based onthe size of the patient's bone, the size of the patient's tendon, e.g.if a tendon autograft is contemplated, the size of the allograft tendon,e.g. if an allograft is contemplated, or the expected size of anartificial graft or the expected biomechanical loads or stresses appliedto the graft; the same or other parameters can also be used in choosinga femoral and/or a tibial anchor for the graft, which can include one ormore interference screws or other types of anchors including button typeanchors. The surgeon or the software can optionally select apredetermined femoral or tibial tunnel location and/or orientation, forexample using the femoral origin of the native, torn ACL as an entrypoint in the femur and the medial tibial spine as an entry point intothe tibia. Note, the term entry and exit point can be usedinterchangeably in the specification.

The projected femoral and/or tibial tunnel location and/or orientationcan be the extension of a line created by connecting the femoral originand tibial insertion of the native ACL, optionally the antero-medialbundle or the postero-lateral bundle or intermediate positions betweenthe two, for example in extension or 15 degrees flexion. The projectedfemoral and/or tibial tunnel location and/or orientation and/or theprojected graft position, location and/or orientation can be determinedfor different flexion and extension and/or rotation angles. If thelocation and/or orientation of the projected femoral and/or tibialtunnel and/or the projected graft varies depending on the degree offlexion, extension and/or rotation, a statistical average can be chosenfor select values or other statistical measures or methods can beapplied to determine the location, position and/or orientation of theprojected femoral and/or tibial tunnel and/or the projected graft.

A graphical user interface, for example implemented on a standard PC orApple computer, can be utilized for displaying the 2D and/or 3D data ofthe patient and for identifying the ACL origin and/or insertion and/orthe ACL remnants as well as any other bony landmarks, features,surfaces, and/or shapes that can be of interest for developing thesurgical plan. The surgeon or the operator can optionally execute thevirtual surgical plan on the graphical user interface. The surgeon orthe operator can place virtual femoral and/or tibial tunnels, e.g. forsingle and for double bundle technique, on the graphical user interfaceand the associated display of the data. The surgeon or the operator canplace virtual grafts, e.g. for single and for double bundle technique,on the graphical user interface and the associated display of the data.The surgeon or the operator can place both virtual tunnels and virtualgrafts on the graphical user interface and the associated display of thedata. The software can optionally display the tunnels and/or the graftin one or more degrees of knee flexion and/or extension and/or rotation.The software and/or the operator can virtually assess the tunnel and/orgraft position, location, and/or orientation for one or more flexion,extension, and/or rotation angles and can perform a virtual assessmentof graft performance for these one or more different angles. Thesoftware and/or the operator/surgeon can optionally make adjustments tothe tunnel and/or graft position, location, and/or orientation based onthe information obtained in this manner from the one or more flexion,extension, and/or rotation angles.

Optionally, the graphical user interface can provide or display anassessment of the mechanical forces applied to the graft and/or theanchor as well as the surrounding bone. Software can be used for thepurpose of assessing the mechanical forces which can, for example,include finite element modeling. In addition, software can be used forassessing the kinematics of the knee for different tunnel and/or graftpositions, locations and/or orientations. Such software can, forexample, include Anybody or other kinematic modeling software.

FIGS. 22E AP and F lateral views demonstrating exemplary virtual ACLtunnels 364 on femoral side and tibial side (straight broken lines).Curved broken line on femoral side indicates intercondylar notcharea/roof.

FIGS. 22G AP and 22H lateral views demonstrating exemplary virtual ACLgraft 366 on femoral side and tibial side extending throughintra-articular space between femur and tibia (straight solid lines).Virtual anchors are also shown on femoral and tibial side (solid blackoval structures) 367. Note, instead of virtual anchors, virtualinterference screws could be used on the femoral and/or the tibial sideor any other means of fixation. Curved broken line on femoral sideindicates intercondylar notch area/roof.

FIG. 23 is an illustrative non-limiting flow chart describing differentapproaches to planning the location, position, orientation, alignmentand/or direction of one or more femoral or tibial tunnels (e.g. forsingle or double bundle technique) or for placing an ACL graft. Scandata can be acquired initially, e.g. ultrasound, CT, MRI 380. The scandata can optionally be segmented 381, e.g. for bone, cartilage, ACLtissue or structures. The segmented 381 or unsegmented 380 scan data canbe displayed in 2D or 3D 382. Optionally, the native ACL origin andinsertion, optionally separate for anteromedial and posterolateralbundle, can be identified 384. Optionally, the native ACL remnants canbe identified, also for anteromedial and posterolateral bundle 386.Optionally, using the information from 384 and/or 386, a graft size 388or tunnel size 390 or both can be selected. Optionally, the virtualfemoral 392 and tibial 396 tunnels can be projected by the OHMD in theirrespective predetermined position and orientation; alternatively, theircentral axis can be projected by the OHMD in its predetermined positionand orientation, all optionally with entry and exit points displayed.Optionally, a virtual ACL graft can be displayed by the OHMD 394 in itspredetermined position. Optionally, steps 392, 394 and/or 396 can beperformed or repeated for different degrees of knee flexion or extensionand/or rotation including instability testing 398. Optionally, thepredetermined position and orientation of the virtual femoral tunnel392, virtual tibial tunnel 396 and/or virtual ACL graft can be checkedin steps 400 and/or 402. Optionally, the predetermined position andorientation of the virtual femoral tunnel 392, virtual tibial tunnel 396and/or virtual ACL graft can be adjusted or modified in steps 404 and/or406.

Optionally, the software can simulate different degrees of femoral andtibial flexion and/or rotation during the range of motion or portions ofthe range of motion.

Registration of Virtual Data and Live Data of the Patient for ACL Repairor Reconstruction

In some embodiments of the invention, the pre-operative imaging or scandata or virtual data of the patient, e.g. from an MRI scan, CT scan,ultrasound scan (2D or 3D), x-ray imaging, or x-ray imaging, ultrasound,CT or MRI with selection of a 3D femoral and/or tibial model of thepatient from a reference database, can be displayed on a computer screenand an operator, e.g. a surgeon or a radiologist, can manually orsemi-automatically identify one or more of the following: lateralfemoral notch wall, ACL origin, proximal ACL remnant(s) on the femoralside, including, for example, antero-medial or postero-lateral bundleportions or intermediate portions, medial tibial spine, distal ACLremnant(s) on the tibial side, including, for example, antero-medial orpostero-lateral bundle portions or intermediate portions, ACL insertionor any other anatomic structure of the knee. The operator, surgeon orradiologist can, for example, click or circle on one more of thesestructures to identify them. Optionally, the operator, surgeon orradiologist can assign a label designating the name of the anatomicstructure that has been identified with the click or circle, e.g.lateral femoral notch wall, ACL origin, proximal ACL remnant(s) on thefemoral side, including, for example, antero-medial or postero-lateralbundle portions or intermediate portions, medial tibial spine, distalACL remnant(s) on the tibial side, including, for example, antero-medialor postero-lateral bundle portions or intermediate portions, ACLinsertion or any other anatomic structure of the knee.

Intra-operatively, the surgeon can then, for example, use a pointer orpointing device to touch the corresponding structures in the live dataof the patient. The pointer or pointing device can be registered inrelationship to an OHMD or a navigation system and/or the patient and/orthe patient's knee, for example with use of one or more IMUs or one ormore optical or navigation markers including infrared markers,retroreflective markers, RF markers, or an image and/or video capturesystem integrated into, attached to or separate from the OHMD so thatthe position of the pointer and the location, position, orientation anddirection of the tip of the pointer is captured in a 3D objectcoordinate system. The surgeon can then optionally touch the structurescorresponding to what was clicked or circled in the pre-operativeimaging/virtual data of the patient in the live data of the patient,i.e. the patient's live knee, as seen, for example, through thearthroscope or with an intraoperative ultrasound probe and scan with apointer. Such structure can, for example, be one or more of a lateralfemoral notch wall, ACL origin, proximal ACL remnant(s) on the femoralside, including, for example, antero-medial or postero-lateral bundleportions or intermediate portions, medial tibial spine, distal ACLremnant(s) on the tibial side, including, for example, antero-medial orpostero-lateral bundle portions or intermediate portions, ACL insertionor any other anatomic structure of the knee. In this manner, virtualdata and live data can be registered in space.

The foregoing anatomic landmarks, surfaces and features are onlyexemplary and are not meant to be limiting. Someone skilled in the artcan readily identify other anatomic landmarks, surfaces or features thatcan be used for purposes of registration of virtual data and live dataof the patient or other data of the patient and/or surgical instruments,for example some of the landmarks in Table 12.

Any of the registration techniques described in the specification can beused for registering virtual data of the patient and live data of thepatient for ACL repair or reconstruction. For example, a pre-operativeimaging test such as an ultrasound scan, CT scan or MRI scan or one ormore x-ray images can be used to produce a patient specific marker. Thepatient specific marker can be designed to have at least one patientspecific surface that can mate with the patient's anatomy, e.g. afemoral surface or a tibial surface. The patient specific marker can beapplied to the patient's femur or tibia. Optionally, the patientspecific marker can be designed so that it can be passed through a smallincision or a small portal inside the knee joint in intra-articularlocation. For this purpose, the patient specific markers can consist ofmultiple parts, which can, optionally, be assembled inside the joint.The sub-parts or components of the patient specific marker can haveengage-able connectors. Once the patient specific marker has beenapplied to the corresponding patient surface(s) and is properly seatedin a mating position, it can optionally be affixed to the underlyingbone or cartilage or ligament structure. The patient specific surface onthe physical patient specific marker which mates with the live patientsurface corresponds to the virtual patient surface in the virtualpatient data. Once the patient specific marker is located in thepredetermined position and orientation on the mating surface in the livepatient, registration between the virtual data and the live data of thepatient can be performed, e.g. using any of the means described in thespecification.

The position of the patient specific marker can optionally be capturedoptically through the arthroscope, for example using an image and/orvideo capture system integrated into or attached to the arthroscopesystem and associated display system. The arthroscope or any relatedinstruments or pointers can be registered in relationship to an OHMD ora navigation system and/or the patient and/or the patient's knee, forexample with use of one or more IMUs or one or more optical ornavigation markers including infrared markers, retroreflective markers,RF markers, or an image and/or video capture system integrated into,attached to or separate from the OHMD so that the position of thearthroscope, instrument and/or pointer and the location, position,orientation and direction of the tip of the arthroscope, instrumentand/or pointer is captured in a 3D object coordinate system that can becross-referenced and registered in relationship to the patient's knee,for example by registering it in relationship to the patient specificmarker and/or in relationship to the OHMD or any other referencecoordinate system used in the operating room.

The patient specific marker can have a known geometric shape, e.g. asquare or a triangle. As the projected shape of the known geometricshape changes in the projection of the arthroscope, the informationabout the change in shape and size of the projected shape of the knowngeometric shape can be used, for example with an image and/or videocapture system, to compute or estimate the position of the arthroscopein relationship to the patient specific marker and/or the live data,e.g. the live arthroscopic images obtained from inside the patient'sjoint, during the procedure. Instead of a known geometric shape, thepatient specific marker can include other markers, e.g. one, two, threeor more LEDs. The change in position of the one, two, three or more LEDsprojected by the arthroscope from within the patient's joint can be usedto compute or estimate the position of the arthroscope in relationshipto the patient specific marker and/or the live data, e.g. the livearthroscopic images obtained from inside the patient's joint, during theprocedure. The patient specific marker can also include physicalreference areas or points, e.g. a groove or a recess that canaccommodate the tip of a pointer. In this manner, the tip of the pointercan be placed in the groove or recess. The pointer can have one or moreIMUs or one or more optical or navigation markers including infraredmarkers, retroreflective markers, RF markers attached to it which can bedetected by the OHMD or a navigation system. The position of the canalso be detected with used of an image and/or video capture system, forexample integrated into, attached to or separate from the OHMD.

In some embodiments of the invention, the patient's knee can be imagedintra-operatively, for example using an x-ray or multiple x-ray imagesor a CT or an ultrasound scan. Anatomic landmarks can be identified onthe scan, which can, for example, include:

TABLE 12: Exemplary anatomic landmarks, surfaces and features in theknee for registration of virtual and live data including, optionally,pre-operative and intraoperative imaging data, for ACLRepair/Reconstruction

Medial wall of the femoral notch

Lateral wall of the femoral notch

Roof of the femoral notch

Residual ACL origin

Residual ACL insertion

Medial wall of the medial condyle

Lateral wall of the lateral condyle

Medial epicondylar eminence

Lateral epicondylar eminence

Medial femoral condyle shape, e.g. radii, convexities, concavities

Lateral femoral condyle shape, e.g. radii, convexities, concavities

Intercondylar notch shape

Intercondylar notch surface features

Medial tibial spine

Lateral tibial spine

Anteromedial tibial rim

Anterolateral tibial rim

Medial tibial rim

Lateral tibial rim

Lowest point of the medial plateau

Lowest point of the lateral plateau

Highest point of the medial plateau

Highest point of the lateral plateau

Medial tibial plateau shape

Lateral tibial plateau shape

Medial tibial plateau surface features, e.g. radii, convexities,concavities

Lateral tibial plateau surface features, e.g. radii, convexities,concavities

The foregoing anatomic landmarks, surfaces and features are onlyexemplary and are not meant to be limiting. Someone skilled in the artcan readily identify other anatomic landmarks, surfaces or features thatcan be used for purposes of registration of virtual data and live dataof the patient or other data of the patient and/or surgical instruments.

The anatomic landmarks, surfaces and features can be used forregistering one or more of the following: pre-operative data, e.g.pre-operative kinematic data, pre-operative imaging data;intra-operative data, e.g. intra-operative kinematic data,intra-operative imaging data; virtual data of the patient, e.g. virtualkinematic data, virtual imaging data, virtual anatomic data, virtualinstrument data, virtual device data, virtual surgical plan of thepatient, live data of the patient including physical surgicalinstruments and arthroscope, for example as seen through the OHMD or ascaptured by an image and/or video capture system integrated into,attached to or separate from the OHMD, or as seen through thearthroscope. The anatomic landmarks, surfaces and features can, forexample, be clicked on or circled or can be identified automatically onone or more of the virtual data of the patient, and/or on one or more ofthe intraoperative imaging data of the patient, e.g. intraoperativelyobtained x-rays or ultrasound, and the corresponding anatomic landmarks,surfaces and features in the live patient/knee can, for example, betouched with a pointer or probe by the surgeon. The pointer or probe canbe registered in relationship to an OHMD or a navigation system and/orthe patient and/or the patient's knee, for example with use of one ormore IMUs or one or more optical or navigation markers includinginfrared markers, retroreflective markers, RF markers or an image and/orvideo capture system integrated into, attached to or separate from theOHMD so that the position of the pointer and the location, position,orientation and direction of the tip of the pointer is captured in a 3Dobject coordinate system which can be cross-referenced to theintra-operative data.

The intra-operative data, e.g. intra-operative imaging data, can bemanually or semi-automatically or automatically (e.g. through imageprocessing and/or pattern recognition techniques) cross-referenced andregistered to the virtual data of the patient, virtual surgical planand/or the live data of the patient. Virtual data, virtual surgicalplan, intra-operative data, e.g. intra-operative imaging, and live dataof the patient can be registered in the same coordinate system,optionally through various coordinate transfers. The surgeon canoptionally touch the structures corresponding to what was clicked orcircled in the pre-operative imaging/virtual data and/or theintra-operative data, e.g. intra-operative imaging data, of the patientin the live data of the patient, i.e. the patient's live knee, as seen,for example, through the arthroscope with a pointer which can include orcarry one or more IMUs or one or more optical or navigation markersincluding infrared markers, retroreflective markers, RF markers or whichcan be registered with use of an image and/or video capture systemintegrated into, attached to or separate from the OHMD. Optionally, anultrasound probe can be introduced through one or more of the portalsand the ultrasound probe can be used for intra-operative imaging, e.g.in addition to x-ray imaging. The ultrasound probe can be used toidentify, for example, the ACL origin, ACL insertion and/or any proximalor distal ACL remnants. The ultrasound probe can include or carry one ormore IMUs or one or more optical or navigation markers includinginfrared markers, retroreflective markers, RF markers which can beregistered with use of an image and/or video capture system integratedinto, attached to or separate from the OHMD. Alternatively, an opticalpointer, e.g. a laser can be used to point at one or more of theanatomic landmarks, surfaces and features in the live patient,corresponding to the anatomic landmarks, surfaces and features that hadbeen marked in the virtual data of the patient and/or theintra-operative data of the patient. The optical pointer can include orcarry one or more IMUs or one or more optical or navigation markersincluding infrared markers, retroreflective markers, RF markers whichcan be registered with use of an image and/or video capture systemintegrated into, attached to or separate from the OHMD or a navigationsystem. Whenever the optical pointer highlights one or more of theanatomic landmarks, surfaces and features in the live patient, the areacan be captured through the imaging system of the arthroscope or throughan an image and/or video capture system integrated into, attached to orseparate from the OHMD. In this manner, corresponding anatomiclandmarks, surfaces and features can be identified in the live data ofthe patient and can be cross-referenced to and registered with thevirtual data of the patient and/or the intra-operative data of thepatient.

The arthroscope, surgical instruments, probes, pointers ACL grafts,femoral and/or tibial anchors and other devices can also be registeredin relationship to any of the anatomic landmarks, surfaces or featuresused for registration in the virtual, intra-operative, live or otherdata of the patient. For this purpose, the physical and, optionally, thevirtual arthroscope, surgical instruments, probes, pointers, ACL grafts,femoral and/or tibial anchors and other virtual devices and/or virtualfemoral and/or tibial tunnels can be registered in relationship to anOHMD or a navigation system and/or the patient and/or the patient'sknee, for example with use of one or more IMUs or one or more optical ornavigation markers including infrared markers, retroreflective markers,RF markers or an image and/or video capture system integrated into,attached to or separate from the OHMD so that the position of thearthroscope, surgical instruments, probes, pointers ACL grafts, femoraland/or tibial anchors and other devices and the location, position,orientation and direction of the arthroscope, surgical instruments,probes, pointers ACL grafts, femoral and/or tibial anchors and otherdevices is captured in a 3D object coordinate system.

Optionally, one or more anatomic landmarks identified on theintraoperative scan can be cross-referenced to the virtual data of thepatient obtained prior to the surgical procedure, e.g. pre-operativex-rays, a CT scan, an MRI scan, or an ultrasound scan, for example usedin developing the virtual surgical plan. The imaging modality usedduring the surgery, e.g. ultrasound, can be different from the imagingmodality used to generate the virtual data of the patient and,optionally, the virtual surgical plan, e.g. an MRI.

Optionally, the arthroscope and/or one or more instruments introducedthrough any of the portals can carry one, two, three or more IMUs,optical, light or other markers, navigation markers including infraredmarkers, retroreflective markers, RF markers, image capture markers(e.g. LEDs) and the like. Only one or more instruments can be registeredin relationship to the virtual data of the patient or theintra-operative data of the patient, while the scope cannot beregistered. Only the scope can be registered in relationship to thevirtual data of the patient or the intra-operative data of the patient,while the one or more instruments cannot be registered. Any markerdescribed in the specification or known in the art can be used. Theposition and/or orientation of the scope and/or the one or moreinstruments can be registered, for example in relationship to one ormore anatomic landmarks identified on the intra-operative imaging dataor in relationship to the virtual data of the patient, e.g. apre-operative x-ray, CT, MRI or ultrasound, or in relationship to thevirtual surgical plan.

Other markers that can be used for any of the foregoing embodiments forACL repair and/or ACL reconstruction include, but are not limited to,skin markers, intra-articular markers, RF markers, optical markers,arthroscopic anchors, arthroscopic tags, pins and/or screws.

In some embodiments of the invention, the surgeon can obtain an image ofthe origin and/or the insertion of the ACL or an image of a proximaland/or a distal remnant of the ACL or a combination of both through thearthroscope or through use of an intraoperative imaging technology suchas an ultrasound, e.g. inserted through one of the portals. A comparableprojection can then be obtained on a computer monitor or in theprojection of the OHMD, wherein the view angle and the magnification ofthe virtual data and the live data of the patient can be substantiallysimilar and can be superimposed, e.g. visually in the OHMD. Oncesubstantial similarity for a view angle and magnification of the livedata and the virtual data of the patient has been obtained, the data canbe registered, e.g. in the same coordinate system or in separatecoordinate systems with a known coordinate transfer. The arthroscope,surgical instruments, probes, pointers, ACL grafts, femoral and/ortibial anchors and other devices can include or can have attached one ormore IMUs or one or more optical or navigation markers includinginfrared markers, retroreflective markers, RF markers or an image and/orvideo capture system can be used that can be integrated into, attachedto or separate from the OHMD so that the arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibial anchorsand other devices can remain registered as they are being moved, forexample after the initial registration using the substantially similarprojections of the physical and the virtual data of the patient.

In some embodiments, the landmarks of the distal femur can beregistered, optionally in relationship to the tibia. The tibia canoptionally be in a fixed position, e.g. with use of a leg holder inrelationship to the femur. Optionally, pins can be placed, e.g. in thebone, e.g. at or near the position of the predetermined femoral tunnel.The position of the one or more pins can be registered, for example withuse of an image and/or video capture system or one or more attached IMUsor optical markers or navigation markers including infrared markers,retroreflective markers, RF markers. In this manner, by keeping the pinand/or one or more IMUs, optical markers or navigation markers in placeor by using an image and/or video capture system integrated into,attached to or separate from the OHMD, the femoral and/or tibialregistration can be maintained even as the knee is moved into differentpositions, e.g. different flexion, extension, rotation, abduction,adduction angles.

In some embodiments, the landmarks of the proximal tibia can beregistered, optionally in relationship to the femur. The femur canoptionally be in a fixed position, e.g. with use of a leg holder inrelationship to the tibia. Optionally, pins can be placed, e.g. in thebone, e.g. at or near the position of the predetermined tibial tunnel.The position of the one or more pins can be registered, for example withuse of an image and/or video capture system or one or more attached IMUsor optical markers or navigation markers. In this manner, by keeping thepin and/or one or more IMUs or optical markers or navigation markers inplace or by using an image and/or video capture system integrated into,attached to or separate from the OHMD, the tibial and/or femoralregistration can be maintained even as the knee is moved into differentpositions, e.g. different flexion, extension, rotation, abduction,adduction angles.

The following data can be registered in relationship to each other usingone or more of the methods described herein:

-   -   Virtual data of the patient, e.g. pre-operative imaging data,        pre-operative kinematic data    -   Virtual surgical plan    -   Intraoperative imaging data, e.g. select landmarks, surfaces or        features of the patient visualized using an intraoperative scan        (see for example foregoing list), e.g. one or more x-rays, CT,        MRI, ultrasound    -   Intraoperative image capture data, e.g. select landmarks,        surfaces or features of the patient's knee (see for example        foregoing list) or the patient's joint    -   One or more patient specific markers applied to the joint, e.g.        applied to one or more articular surfaces or osteophytes,        optionally visualized using an image and/or video capture system        integrated into, attached to or separate from the arthroscopy        system or optionally visualized through the arthroscopy system    -   Scope position, location, orientation, alignment and direction,        for example measured via attached IMUs, optical markers,        navigation markers including infrared markers, retroreflective        markers, RF markers, or an image and/or video capture system        integrated into, attached to or separate from the OHMD    -   Instrument, probe, graft, anchor or other device position,        location, orientation, alignment and direction, for example        measured via attached IMUs, optical markers, navigation markers        including infrared markers, retroreflective markers, RF markers,        or an image and/or video capture system integrated into,        attached to or separate from the OHMD

Projected Path of the Physical Instruments, Devices or Grafts andVirtual Path of Virtual Instruments, Devices, Grafts or Tunnels

Registration can be effected or achieved using any of the techniquesdescribed in the specification. For example, the position, location,orientation, direction of any of the IMUs or optical markers ornavigation markers including infrared markers, retroreflective markers,RF markers, integrated into or attached to any of the arthroscope,surgical instruments, probes, pointers, ACL grafts, femoral and/ortibial anchors and other devices can be captured using the OHMD ornavigation system or the position, location, orientation, direction ofthe arthroscope, surgical instruments, probes, pointers, ACL grafts,femoral and/or tibial anchors and other devices can be captured using animage and/or video capture system, e.g. integrated into, attached to orseparate from the OHMD, and, for example, a projected path for anphysical surgical instrument, e.g. a probe or a drill, can be computedand/or displayed by the OHMD and/or the display monitor(s) of thearthroscopy system once registration has been completed. The projectedpath of an physical surgical instrument can, for example, be parallelto, coinciding with, superimposed onto, or orthogonal to or at a definedangle to the predetermined position and/or orientation of one or more ofthe predetermined femoral tunnel, tibial tunnel, ACL graft or anchor(s)or interference screws. The projected path can change as the position,location, orientation, direction of the physical arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibial anchorsand other devices changes. The projected path can be an extension of thelong or other axis or the direction of travel of the one or more of thephysical arthroscope, surgical instruments, probes, pointers, ACLgrafts, femoral and/or tibial anchors and other devices. The projectedpath can be displayed by the OHMD, in 3D stereoscopic or 3Dnon-stereoscopic or 2D form, optionally with different colors orpatterns. The projected path can be displayed by the display monitor ofthe arthroscopy system and/or both.

If the display is through the OHMD, the magnification can be adjusted ifthe operator looks at the patient or the patient's knee, which canrequire, for example, no magnification, or if the operator looks at thedisplay of the arthroscopic images obtained through the scope and,optionally, displayed by the monitor system of the arthroscopy unit.Since images of the patient's knee and internal structures obtainedthrough the arthroscope and optionally displayed by the arthroscopysystem display monitor are typically magnified, the display of theprojected path can be magnified as well, for example matching themagnification factor of the arthroscopy display or system. The displayof the projected path and/or any virtual instruments or virtual displaysof any non-visualized portions of physical instruments can be matched inmagnification to the magnification of the arthroscopic images or theinherent magnification of the arthroscopy system or, optionally, it canbe slightly less or more in magnification than the magnification of thearthroscope or the arthroscopy monitor display unit.

In some embodiments of the invention, a virtual path for one or more thearthroscope, surgical instruments, probes, pointers, ACL grafts, femoraland/or tibial anchors and other devices can be displayed by the OHMDand/or the arthroscopy system display monitor(s). The virtual path cancoincide or be substantially aligned with or parallel with or beidentical with the predetermined path of the surgical instrument. Thevirtual path can, for example, be parallel to, coinciding with,superimposed onto, or orthogonal to or at a defined angle to thepredetermined position and/or orientation of one or more of thepredetermined femoral tunnel, tibial tunnel, ACL graft or anchor(s) orinterference screws. The virtual path can be projected through the OHMD,optionally in 3D stereoscopic or 3D non-stereoscopic or 2D form,optionally with different colors or patterns. The virtual path can beprojected by the display monitor of the arthroscopy system. Virtualinstruments and or devices such as virtual arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibial anchorsand other virtual devices can also be displayed by the OHMD and/or thearthroscopy system display monitor(s).

If the display is through the OHMD, the magnification can be adjusted ifthe operator looks at the patient or the patient's knee, which canrequire, for example, no magnification, or if the operator looks at thedisplay of the arthroscopic images obtained through the scope and,optionally, displayed by the display monitor system of the arthroscopyunit. Since images of the patient's knee and internal structuresobtained through the arthroscope and optionally displayed by thearthroscopy system display monitor are typically magnified, the displayof the virtual path or any virtual instruments or devices such as thearthroscope, surgical instruments, probes, pointers, ACL grafts, femoraland/or tibial anchors and other virtual devices can be magnified aswell, for example matching the magnification factor of the arthroscopydisplay or system. The display of the virtual path and/or any virtualinstruments or devices can be matched in magnification to themagnification of the arthroscopic images or the inherent magnificationof the arthroscopy system or, optionally, it can be slightly less ormore in magnification than the magnification of the arthroscope or thearthroscopy monitor display unit. The display of the virtual path and/orany virtual instruments or devices can be using different colors orpatterns, for example different than the live data of the patient,including the arthroscopic images of the internal structures of theknee.

In some embodiments of the invention, the extremity, in the case ofshoulder surgery or elbow surgery the arm, in the case of knee or hipsurgery, including ACL repair or reconstruction, is held or positionedin the same position, e.g. the same degrees of flexion, extension,abduction, adduction, internal or external rotation, for the acquisitionof data that will be used for purposes of registration of pre-operativedata, e.g. pre-operative imaging data and/or kinematic data,intra-operative data, e.g. intra-operative imaging data and/or kinematicdata, and/or live data of the patient, e.g. data observed through theOHMD such as live patient data of the knee joint or data observedthrough the OHMD or the display monitor unit of the arthroscopy system,e.g. live patient data of the internal structures of the patient's knee.By acquiring these pre-operative, intra-operative and live patient datain the same position of the extremity or the target tissue or the joint,less variability in positioning can be encountered which can helpfacilitate registration using any of the methods described in thespecification. For example, an upper arm holder or a leg holder can beused for obtaining pre-operative imaging data, e.g. x-ray images,ultrasound data, CT or MRI data of the extremity or target joint ortarget tissue; the upper arm or leg holder can fixate the extremity ortarget joint or target tissue in one or more positions. The same or asimilar upper arm holder or a leg holder can be used for obtainingintra-operative imaging data, e.g. x-ray images, ultrasound data, CT orMRI data of the extremity or target joint or target tissue; the upperarm or leg holder can fixate the extremity or target joint or targettissue in one or more positions for the intra-operative dataacquisition. The live patient data including arthroscopic data obtainedfrom inside the patient's joint can be obtained with the extremity, thetarget joint or the target tissue in a similar position than that usedwhen the pre-operative or intra-operative data were obtained.Registration of two or more of pre-operative data of the patient,intra-operative data of the patient, virtual data of the patient,virtual surgical plan of the patient or live data of the patient,including arthroscopic image or other data obtained from within thepatient's joint can be facilitated in this manner.

Any of the foregoing embodiments, e.g. those related to virtual surgicalplans, registration, and extremity or target joint or tissue positioningare applicable to other surgical procedures, e.g. knee replacement, hipreplacement, spinal surgery, spinal fusion, vertebroplasty, kyphoplasty,brain surgery, other organ surgery, e.g. liver, renal, spleen,intestinal surgery as well as removal of any kind of neoplasms.

OHMD

The OHMD can optionally display the one or more virtual tunnels or thevirtual graft or the virtual graft position. The OHMD can also display aprojected path of one or more physical surgical instruments or devices,e.g. an arthroscope, surgical instruments, probes, pointers, ACL grafts,femoral and/or tibial anchors and other devices. The OHMD can alsodisplay the virtual path of one or more physical surgical instruments ordevices, e.g. an arthroscope, surgical instruments, probes, pointers,ACL grafts, femoral and/or tibial anchors and other devices. The virtualpath can be the predetermined path from the virtual surgical plan.

Optionally, one or more physical surgical instrument(s) or devices, e.g.an arthroscope, probes, pointers, ACL grafts, femoral and/or tibialanchors and other devices, and/or their projected path can be alignedwith the display of the virtual tunnel(s), virtual graft or virtualgraft position. Alternatively, the OHMD can display the virtual positionof the corresponding virtual surgical instrument(s) or devices, e.g. anarthroscope, surgical instruments, probes, pointers, ACL grafts, femoraland/or tibial anchors and other devices, and the operator can optionallyalign the one or more physical surgical instruments or devices with thevirtual surgical instruments or devices. Alternatively, the OHMD candisplay the position and/or orientation and/or alignment or direction oftravel of the virtual surgical instruments or devices as well as one ormore of the virtual tunnel(s), virtual graft or virtual graft positionand the physical surgical instruments or devices, e.g. a probe or drill,and/or their projected path can be aligned with combinations of both ofthe virtual surgical instruments or devices and or the virtualtunnel(s), virtual graft or virtual graft.

The projected path, virtual path, predetermined path, virtual surgicalinstruments and/or devices, the arthroscope, surgical instruments,probes, pointers, ACL grafts, femoral and/or tibial anchors and othervirtual devices, virtual tunnel(s), and/or virtual graft can bedisplayed by the OHMD and/or the display unit of the arthroscopy systemusing different patterns and colors, e.g. solid lines, broken lines,dotted lines, different colors, e.g. green, red, blue, orange, differentthickness, different opacity or transparency.

In some embodiments, one or more IMUs and/or optical markers, LEDs,navigation markers including infrared markers, retroreflective markers,RF markers, calibration phantoms can be applied to the arthroscope,surgical instruments, probes, pointers, ACL grafts, femoral and/ortibial anchors and other devices. The arthroscope, surgical instruments,probes, pointers, ACL grafts, femoral and/or tibial anchors and otherdevices can be registered, e.g. in relationship to the virtual data ofthe patient. The arthroscope or one or more arthroscope instruments,e.g. probes or pointers, can be applied to various landmarks inside theknee joint, visualized through the arthroscope, and registered with thepatient's virtual data, e.g. preo-operative scan data, and/orintra-operative scan data.

Once virtual data and live data of the patient are registered, thephysical drill or instrument used for preparing the tunnel can bealigned with the axis, position and/or orientation of the virtual drillor virtual tunnel displayed by the OHMD and/or the display unit of thearthroscopy system, both on the femoral and on the tibial side.Alternatively, a virtual path can be displayed by the OHMD and/or thedisplay unit of the arthroscopy system, and the physical drill, e.g. thelong axis of the physical drill, and the entry point of the physicaldrill can be aligned with the virtual path. This can be performed withsingle and double bundle technique. This can also be performedseparately for the femoral and/or the tibial tunnel and the femoraland/or the tibial side of the graft. If a transtibial technique is used,the femoral and the tibial tunnels can be linked for a given angle ofknee flexion (and/or rotation) in the virtual surgical plan and thevirtual display by the OHMD or the display unit of the arthroscopysystem so that the virtual surgical plan is consistent with the intendedtranstibial technique of the surgeon.

In some embodiments of the invention, the scope can optionally have oneor more IMUs or optical markers or navigation markers including infraredmarkers, retroreflective markers, RF markers attached and the scope canbe registered in its location in relationship to the OHMD. The positionand/or orientation of the scope can also be captured with an imageand/or video capture system integrated into, attached to or separatefrom the OHMD. The surgeon can move the arthroscope back and forth overa target area of the distal femur or the proximal tibia, e.g. thearea(s) of the approximate tunnel placement or the area(s) of the ACLorigin and/or insertion. By moving the scope back and forth over thetarget area, a visual perception of the surface topography and/or shapecan be obtained. In addition, since the scope can be registered in acoordinate system with use of the one or more IMUs, optical markers,navigation markers including infrared markers, retroreflective markers,RF markers and/or the image and/or video capture system, the surfacetopography and/or shape of the target area can also be captured andregistered in relationship to the scope and/or the OHMD and theirrespective object coordinate systems. As the scope is moved back andforth over the target area, multiple projections of the target area canbe obtained by the scope at different angular orientations of the scope.Optionally, these multiple angular projections of the target area can beused to reconstruct a 3D surface of the target area or estimate a targetarea surface or topography or shape from the scope image data. Thesurface topography and/or the shape can be compared to the surfacetopography and/or shape of the target area in the virtual data of thepatient or, optionally, the intra-operative data of the patient. Usingstandard image processing techniques known in the art and featurecomparisons, substantially similar surface topographies and/or shapescan be identified in the scope image data and the virtual data of thepatient and registration of the scope image data and with that live dataof the patient, virtual data of the patient, and/or OHMD can beperformed. Any object coordinate transfers can now be known for purposesof the registration.

If the virtual surgical instruments, devices, grafts or tunnels and/orthe virtual surgical plan are displayed by the OHMD and/or the displayunit of the arthroscopy system, the surgeon can move the arthroscopeback and forth or, for example, in a circular fashion to obtain a depthperspective or pseudo 3D effect of the intra-articular structuresincluding, for example, the visual representation by the arthroscopyunit of the respective tunnel entry areas; while the surgeon is movingthe arthroscope in this manner, the arthroscope motion including thechange in angular orientation or direction can be monitored using one ormore IMUs or optical markers and/or navigation markers includinginfrared markers, retroreflective markers, RF markers attached to thearthroscope or it can be monitored by an image and/or video capturesystem integrated into, attached to or separate from the OHMD. Thesoftware can maintain the registration of the arthroscopy system inrelationship to the virtual data of the patient and the live data of thepatient through the change in angular orientation and or direction andthe display of the virtual surgical instruments, devices, grafts ortunnels and/or the virtual surgical plan will remain steady in the OHMD.

The following is an exemplary list of physical and virtual instrumentsthat can be used during the ACL reconstruction. The OHMD can display oneor more or all of these instruments in virtual form during the course ofthe surgical procedure following the virtual surgical plan. For example,each virtual instrument can be displayed with the predeterminedposition, location, orientation and/or direction to execute on thevirtual surgical plan so that the surgeon can align the physicalinstruments used for the ACL reconstruction with the virtual instrumentsdisplayed by the OHMD or the virtual tunnels on the femoral or tibialside or a central tunnel axis on the femoral or the tibial side or avirtual ACL graft. The OHMD can also display a virtual path of thevirtual surgical instruments, wherein the virtually path can be thepredetermined path from the virtual surgical plan. The OHMD can alsodisplay the projected path of the physical surgical instruments used inthe live data of the patient.

TABLE 13: Exemplary list of physical surgical instruments and virtualsurgical instruments displayed by the OHMD for ACL reconstruction(multiple of each can be used, e.g. with different dimensions, lengths,shapes):

Arthroscope

Power instrument

Power tool

Arthroscopy portal

Arthroscopy sheath

Obturator

Grasper, e.g. alligator grasper, bulldog grasper

Duckster

Upbiter

Punch, e.g. wishbone punch

Burr

Shaver

Suture cutter

Scissors

Drill, various kinds, different diameters, solid, cannulated

Drill guide

Offset drill guide

Offset drill guide screw or pin

Drill sleeve(s), various kinds, e.g. stepped drill sleeve, straightdrill sleeve

Chuck key

Hook probe

Parallel guide

Parallel guide sleeve

Tendon stripper, e.g. semitendinosus tendon stripper, hamstring tendonstripper

Rasp, e.g. notchplasty rasp

Reamer

Reamer handle

Pin puller

Tunnel plug

Tunnel notcher

Retractor, e.g. graft harvesting retractor

Hook, e.g. femoral or tibial ACL marking hook

ACL guide, e.g. femoral or tibial, left, right

Screwdriver, e.g. retro-screwdriver, interference screw driver

Screwdriver shaft, cannulated or non-cannulated

ACL guides, e.g. transportal ACL guide

ACL drill guide, e.g. transtibial ACL drill guide

Tibial or femoral tunnel guides, e.g. with single point elbow slide,single point forked slide, dual point forked slide

Femoral aimer

Tibial aimer

Osteotome

Handle

Cannula, e.g. tibial tunnel cannula, optionally with one or moreobturators

Cut guides, e.g. for graft harvesting

Graft sizing tool

Graft knife

Graft knife holder/handle

Interference screw, resorbable, non-resorbable

The OHMD can display the complete femoral and/or tibial tunnel, it candisplay only a central line or axis, or it can display a directionalarrow for the tunnel(s). The OHMD can display the complete femoraland/or intra-articular and/or tibial graft portions, it can display onlya central line or axis in the femoral, intra-articular or tibial area,or it can display a directional arrow for the graft(s).

Optionally, if the surgeon elects to change the physical surgical plan,the virtual surgical plan can be adapted accordingly, for example via acomputer interface, and the sequence of steps and virtual instrumentsdisplayed in the OHMD can be changed by changing the virtual surgicalplan. Changes to the virtual surgical plan can include a change insequence of surgical steps, a change in surgical approach, e.g. femurfirst, tibia first, transtibial, omitting select surgical steps, addingsurgical steps, re-orienting virtual surgical tunnel(s), re-orientingvirtual surgical graft etc.

If the surgeon elects to adjust the position, location and/ororientation of the femoral or the tibial tunnel, the software can adjustthe position of the tunnel on the opposing side in the virtual surgicalplan. Such adjustments can be automatic, e.g. if a transtibial techniqueis used, the femoral tunnel can be an extension of an adjusted tibialtunnel. The adjustments in the virtual surgical plan of the opposingtunnel can also be manual, e.g. by the surgeon, for example after thesurgeon has adjusted the first physical tunnel and altered its positionin relationship to the virtual surgical plan. The software canoptionally re-compute the location of the opposing tibial tunnel fordifferent angles of extension, flexion and rotation after the positionand/or orientation of the first tunnel has been changed, either in thevirtual surgical plan or in the physical surgery.

Any surgical technique or approach known in the art for ACLreconstruction and also for ACL repair can be used. Accordingly, virtualsurgical plans can be used for any surgical technique or approach knownin the art for ACL reconstruction and also for ACL repair and can bedisplayed by the OHMD. Such surgical techniques or approaches caninclude, but are not limited to, open surgical ACL reconstruction orrepair, arthroscopic surgical ACL reconstruction or repair, all insideACL reconstruction or repair, trans-tibial ACL reconstruction, femurfirst techniques, tibia first techniques, use of interference screws orother types of anchors, single and double bundle techniques, patellarautograft techniques, semitendinosus tendon techniques, other types oftendon graft techniques, allograft techniques.

In some embodiments of the invention, the OHMD can optionally displayany non-visualized portions of one or more of the physical arthroscope,surgical instruments, probes, pointers, ACL grafts, femoral and/ortibial tunnels, femoral and/or tibial anchors and other devices. Sincethe geometries, shapes and dimensions of the physical arthroscope,surgical instruments, probes, pointers, ACL grafts, femoral and/ortibial anchors and other devices are known, optionally an image and/orvideo capture system can be used to capture the visualized portions ofthe one or more arthroscope, surgical instruments, probes, pointers, ACLgrafts, femoral and/or tibial anchors and other devices. Optionalmarkers, e.g. cm or mm marks, can be used to identify which portions ofthe physical arthroscope, surgical instruments, probes, pointers, ACLgrafts, femoral and/or tibial anchors and other devices are visualizedand which ones are not visualized. The software can then identify whichportions of the arthroscope, surgical instruments, probes, pointers, ACLgrafts, femoral and/or tibial tunnels, femoral and/or tibial anchors andother devices are not included in the image capture data or are notvisualized and the software can compute the position, location,orientation and size/magnification (if applicable) of the non-visualizedportions of the arthroscope, surgical instruments, probes, pointers, ACLgrafts, femoral and/or tibial tunnels, femoral and/or tibial anchors andother devices, which can then optionally be displayed by the OHMD, e.g.as an extension of the visualized portions of the physical arthroscope,surgical instruments, probes, pointers, ACL grafts, femoral and/ortibial tunnels, femoral and/or tibial anchors and other devices. Othermeans described in the specification for displaying the non-visualizedportions of the physical arthroscope, surgical instruments, probes,pointers, ACL grafts, femoral and/or tibial tunnels, femoral and/ortibial anchors and other devices can be used. The non-visualizedportions of the physical arthroscope, surgical instruments, probes,pointers, ACL grafts, femoral and/or tibial tunnels, femoral and/ortibial anchors and other devices can be displayed by the OHMD, thedisplay unit of the arthroscopy system or both.

In some embodiments of the invention, the OHMD can display one or moreprojected paths for one or more physical arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibialtunnels, femoral and/or tibial anchors and other devices and/or it candisplay one or more virtual arthroscope, surgical instruments, probes,pointers, ACL grafts, femoral and/or tibial tunnels, femoral and/ortibial anchors and other devices and/or one or more virtual paths and/orvirtual surgical plans for the one or more virtual arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibialtunnels, femoral and/or tibial anchors and other devices. In someembodiments, the display unit of the arthroscopy system, e.g. one ormore electronic monitors used, can display one or more projected pathsfor one or more physical arthroscope, surgical instruments, probes,pointers, ACL grafts, femoral and/or tibial tunnels, femoral and/ortibial anchors and other devices and/or it can display one or morevirtual arthroscope, surgical instruments, probes, pointers, ACL grafts,femoral and/or tibial tunnels, femoral and/or tibial anchors and otherdevices and/or one or more virtual paths and/or virtual surgical plansfor the one or more virtual arthroscope, surgical instruments, probes,pointers, ACL grafts, femoral and/or tibial tunnels, femoral and/ortibial anchors and other devices. In some embodiments s of theinvention, both the OHMD and the display unit of the arthroscopy system,e.g. one or more electronic monitors used, can display one or moreprojected paths for one or more physical arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibialtunnels, femoral and/or tibial anchors and other devices and/or it candisplay one or more virtual arthroscope, surgical instruments, probes,pointers, ACL grafts, femoral and/or tibial tunnels, femoral and/ortibial anchors and other devices and/or one or more virtual paths and/orvirtual surgical plans for the one or more virtual arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibialtunnels, femoral and/or tibial anchors and other devices.

The OHMD can display the one or more projected paths for one or morephysical arthroscope, surgical instruments, probes, pointers, ACLgrafts, femoral and/or tibial tunnels, femoral and/or tibial anchors andother devices and/or it can display the one or more virtual arthroscope,surgical instruments, probes, pointers, ACL grafts, femoral and/ortibial tunnels, femoral and/or tibial anchors and other devices and/orone or more virtual paths and/or virtual surgical plans for the one ormore virtual arthroscope, surgical instruments, probes, pointers, ACLgrafts, femoral and/or tibial tunnels, femoral and/or tibial anchors andother devices using a magnification or size that is reflective of orcorresponds to the distance of the OHMD or the surgeon's or operator'seyes to the patient's knee joint when the surgeon looks at the kneejoint. The display unit, e.g. one or more electronic monitors, of thearthroscopy system can display the one or more projected paths for oneor more physical arthroscope, surgical instruments, probes, pointers,ACL grafts, femoral and/or tibial tunnels, femoral and/or tibial anchorsand other devices and/or it can display the one or more virtualarthroscope, surgical instruments, probes, pointers, ACL grafts, femoraland/or tibial tunnels, femoral and/or tibial anchors and other devicesand/or one or more virtual paths and/or virtual surgical plans for theone or more virtual arthroscope, surgical instruments, probes, pointers,ACL grafts, femoral and/or tibial tunnels, femoral and/or tibial anchorsand other devices using a magnification or size that is reflective of orcorresponds to the magnification of the display unit of the arthroscopysystem for the display of the live data of the patient from inside thepatient's knee joint so that the size and/or magnification of the one ormore projected paths for one or more physical arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibialtunnels, femoral and/or tibial anchors and other devices and/or the oneor more virtual arthroscope, surgical instruments, probes, pointers, ACLgrafts, femoral and/or tibial tunnels, femoral and/or tibial anchors andother devices and/or one or more virtual paths and/or virtual surgicalplans for the one or more virtual arthroscope, surgical instruments,probes, pointers, ACL grafts, femoral and/or tibial tunnels, femoraland/or tibial anchors and other devices is matched to the live displayof the intra-articular structures of the patient's knee joint visualizedby the arthroscopy system. In this manner, the surgeon can work in aseamless manner between live intra-articular image data of the patientand projected data and virtual data of the patient since they can havematching size and/or magnification.

The magnification used by the OHMD for displaying the one or moreprojected paths for one or more physical arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibialtunnels, femoral and/or tibial anchors and other devices and/or one ormore virtual arthroscope, surgical instruments, probes, pointers, ACLgrafts, femoral and/or tibial tunnels, femoral and/or tibial anchors andother devices and/or one or more virtual paths and/or virtual surgicalplans for one or more virtual arthroscope, surgical instruments, probes,pointers, ACL grafts, femoral and/or tibial tunnels, femoral and/ortibial anchors and other devices can change as the surgeon moves closerto or further away from the patient's knee. The magnification of thedisplay unit of the arthroscopy system can change; for example, it canbe increased or decreased, and the magnification for displaying the oneor more projected paths for one or more physical arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibialtunnels, femoral and/or tibial anchors and other devices and/or one ormore virtual arthroscope, surgical instruments, probes, pointers, ACLgrafts, femoral and/or tibial tunnels, femoral and/or tibial anchors andother devices and/or one or more virtual paths and/or virtual surgicalplans for one or more virtual arthroscope, surgical instruments, probes,pointers, ACL grafts, femoral and/or tibial tunnels, femoral and/ortibial anchors and other devices can be adjusted correspondingly.

The OHMD can optionally display the one or more projected paths for oneor more physical arthroscope, surgical instruments, probes, pointers,ACL grafts, femoral and/or tibial tunnels, femoral and/or tibial anchorsand other devices and/or one or more virtual arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibialtunnels, femoral and/or tibial anchors and other devices and/or one ormore virtual paths and/or virtual surgical plans for one or more virtualarthroscope, surgical instruments, probes, pointers, ACL grafts, femoraland/or tibial tunnels, femoral and/or tibial anchors and other deviceswhen the surgeon looks at the display unit of the arthroscopy systemthrough the OHMD using a magnification or size that is reflective of orcorresponds to or is larger or smaller than the magnification used bythe display unit of the arthroscopy system for the display of the livedata from inside the patient's knee joint.

The magnification used by the OHMD for the display of the one or moreprojected paths for one or more physical arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibialtunnels, femoral and/or tibial anchors and other devices and/or one ormore virtual arthroscope, surgical instruments, probes, pointers, ACLgrafts, femoral and/or tibial tunnels, femoral and/or tibial anchors andother devices and/or one or more virtual paths and/or virtual surgicalplans for one or more virtual arthroscope, surgical instruments, probes,pointers, ACL grafts, femoral and/or tibial tunnels, femoral and/ortibial anchors and other devices can switch back to be reflective of orcorrespond to the distance of the OHMD or the surgeon's eyes to thepatient's knee or it can be smaller or larger when the surgeon looks atthe patient's knee again, rather than the display unit of thearthroscopy system.

In some embodiments, the display unit of the arthroscopy system candisplay the one or more projected paths for one or more physicalarthroscope, surgical instruments, probes, pointers, ACL grafts, femoraland/or tibial tunnels, femoral and/or tibial anchors and other devicesand/or one or more virtual arthroscope, surgical instruments, probes,pointers, ACL grafts, femoral and/or tibial tunnels, femoral and/ortibial anchors and other devices and/or one or more virtual paths and/orvirtual surgical plans for one or more virtual arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibialtunnels, femoral and/or tibial anchors and other devices. The displayunit of the arthroscopy system, e.g. one or two electronic monitors, candisplay the one or more projected paths for one or more physicalarthroscope, surgical instruments, probes, pointers, ACL grafts, femoraland/or tibial tunnels, femoral and/or tibial anchors and other devicesand/or one or more virtual arthroscope, surgical instruments, probes,pointers, ACL grafts, femoral and/or tibial tunnels, femoral and/ortibial anchors and other devices and/or one or more virtual paths and/orvirtual surgical plans for one or more virtual arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibialtunnels, femoral and/or tibial anchors and other devices at amagnification that is reflective of or corresponds to the magnificationof the live data of the structures projected from inside the patient'sknee seen through the arthroscope and displayed by the display unit ofthe arthroscopy unit, or at a magnification that is smaller or largerthan that.

When the surgeon looks through the OHMD at the display unit of thearthroscopy system, e.g. one or two electronic monitors, the OHMD canoptionally turn of the display of the one or more projected paths forone or more physical arthroscope, surgical instruments, probes,pointers, ACL grafts, femoral and/or tibial tunnels, femoral and/ortibial anchors and other devices and/or one or more virtual arthroscope,surgical instruments, probes, pointers, ACL grafts, femoral and/ortibial tunnels, femoral and/or tibial anchors and other devices and/orone or more virtual paths and/or virtual surgical plans for one or morevirtual arthroscope, surgical instruments, probes, pointers, ACL grafts,femoral and/or tibial tunnels, femoral and/or tibial anchors and otherdevices. The turning off or turning on of the display of the one or moreprojected paths for one or more physical arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibialtunnels, femoral and/or tibial anchors and other devices and/or one ormore virtual arthroscope, surgical instruments, probes, pointers, ACLgrafts, femoral and/or tibial tunnels, femoral and/or tibial anchors andother devices and/or one or more virtual paths and/or virtual surgicalplans for one or more virtual arthroscope, surgical instruments, probes,pointers, ACL grafts, femoral and/or tibial tunnels, femoral and/ortibial anchors and other devices can be performed via manual commands,voice commands, various commands from various input systems, orautomatically. An automatic turning on or off can be achieved, forexample, with use of an image and/or video capture system integratedinto, attached to or separate from the OHMD. The image and/or videocapture system can, for example, capture the outline of the display unitof the arthroscopy system and the software can the automatically turnoff the OHMD display or aspects of the OHMD display. Alternatively, thedisplay unit of the arthroscopy system can have one or more markers,e.g. one or more LEDs, that the image and/or video capture system candetect which, in turn, can then trigger the turning on or off of theOHMD display.

In some embodiments, the OHMD can detect, e.g. automatically, if thesurgeon or operator is looking at the display unit of the arthroscopysystem, for example, with use of an image and/or video capture systemintegrated into, attached to or separate from the OHMD. The image and/orvideo capture system can, for example, capture the outline of thedisplay unit of the arthroscopy system and the software can theautomatically adjust the magnification of the items displayed by theOHMD so that it is reflective of, corresponds to, is smaller or largerthan the magnification used by the display unit of the arthroscopysystem for the live data/images from inside the patient's knee.Alternatively, the display unit of the arthroscopy system can have oneor more markers, e.g. one or more LEDs, that the image and/or videocapture system can detect which, in turn, can then trigger theadjustment of the magnification of the items displayed by the OHMD.

Similarly, the OHMD can detect, e.g. automatically, if the surgeon oroperator is not looking at the display unit of the arthroscopy system,for example, with use of an image and/or video capture system integratedinto, attached to or separate from the OHMD. The image and/or videocapture system can, for example, detect that the outline of the displayunit of the arthroscopy system is not present in the captured image dataand the software can the automatically adjust the magnification of theitems displayed by the OHMD so that it is reflective of or correspondsto the distance of the OHMD or the surgeon's eyes to the patient's knee,or is smaller or larger than that. Alternatively, the display unit ofthe arthroscopy system can have one or more markers, e.g. one or moreLEDs, that the image and/or video capture system can detect; in thiscase, when the image captures system notices that the one or more LEDsare not included in the image capture data, the software can theautomatically adjust the magnification of the items displayed by theOHMD so that it is reflective of or corresponds to the distance of theOHMD or the surgeon's eyes to the patient's knee, or is smaller orlarger than that. Similarly, markers or LEDs placed on the patient'sknee can be detected by the OHMD including an image and/or video capturesystem integrated into, attached to or separate from the OHMD therebytriggering an adjustment in magnification so that it is reflective of,corresponds to the distance of the OHMD or the surgeon's eyes to thepatient's knee, or is smaller or larger than that when the surgeon oroperator is looking at the patient's knee.

In some embodiments of the invention, the OHMD can be used to displaythe live data collected by the arthroscope from inside the patient'sknee, for example instead of the display unit of the arthroscopy systemor in addition to the display unit of the arthroscopy system.Optionally, the OHMD can replace the display unit of the arthroscopysystem or it can be the display unit of the arthroscopy system. In thisexample, the OHMD can display the live data from inside the patient'sknee collected by the arthroscope and project them for the surgeon. TheOHMD can also display one or more projected paths for one or morephysical arthroscope, surgical instruments, probes, pointers, ACLgrafts, femoral and/or tibial tunnels, femoral and/or tibial anchors andother devices and/or one or more virtual arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibialtunnels, femoral and/or tibial anchors and other devices and/or one ormore virtual paths and/or virtual surgical plans for one or more virtualarthroscope, surgical instruments, probes, pointers, ACL grafts, femoraland/or tibial tunnels, femoral and/or tibial anchors and other devicesin addition to the live images from inside the patient's knee. In thisembodiment, the OHMD can optionally match the magnification of the oneor more projected paths for one or more physical arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibialtunnels, femoral and/or tibial anchors and other devices and/or one ormore virtual arthroscope, surgical instruments, probes, pointers, ACLgrafts, femoral and/or tibial tunnels, femoral and/or tibial anchors andother devices and/or one or more virtual paths and/or virtual surgicalplans for one or more virtual arthroscope, surgical instruments, probes,pointers, ACL grafts, femoral and/or tibial tunnels, femoral and/ortibial anchors and other devices relative to the magnification of thelive data from inside the patient's knee collected by the arthroscope.The OHMD can also can apply a larger or smaller magnification and orsize than the magnification of the live data from inside the patient'sknee collected by the arthroscope for the one or more projected pathsfor one or more physical arthroscope, surgical instruments, probes,pointers, ACL grafts, femoral and/or tibial tunnels, femoral and/ortibial anchors and other devices and/or one or more virtual arthroscope,surgical instruments, probes, pointers, ACL grafts, femoral and/ortibial tunnels, femoral and/or tibial anchors and other devices and/orone or more virtual paths and/or virtual surgical plans for one or morevirtual arthroscope, surgical instruments, probes, pointers, ACL grafts,femoral and/or tibial tunnels, femoral and/or tibial anchors and otherdevices.

In some embodiments of the invention, for example when the OHMD is theprimary display unit of the arthroscopy system, the OHMD can benon-transparent to light or minimally transparent to light reflectedfrom the patient's knee or surgical theatre and can display, forexample, live (electronic) images collected by the arthroscope fromwithin the patient's knee and, optionally, it can display, in addition,one or more projected paths for one or more physical arthroscope,surgical instruments, probes, pointers, ACL grafts, femoral and/ortibial tunnels, femoral and/or tibial anchors and other devices and/orone or more virtual arthroscope, surgical instruments, probes, pointers,ACL grafts, femoral and/or tibial tunnels, femoral and/or tibial anchorsand other devices and/or one or more virtual paths and/or virtualsurgical plans for one or more virtual arthroscope, surgicalinstruments, probes, pointers, ACL grafts, femoral and/or tibialtunnels, femoral and/or tibial anchors and other devices (with variouschosen matching or non-matching magnifications). In this setting, theOHMD can also display electronic images of the physical arthroscope,surgical instruments, probes, pointers, ACL grafts, femoral and/ortibial tunnels, femoral and/or tibial anchors and other devices andtheir respective movements, for example captured with an image and/orvideo capture system integrated into, attached to, or separate from theOHMD (with various chosen matching or non-matching magnifications).

The OHMD can be permanently non-transparent to light or minimallytransparent to light reflected from the patient's knee or surgicaltheatre. Alternatively, the degree of transparency can be variable, forexample with use of one or more optical filters, e.g. polarizing lightfilters, in front of or integrated into the OHMD or electronic, e.g.LCD, or optical filters in front or integrated into the OHMD. The ORtheater can optionally use light sources, e.g. polarized or filteredlight that will support modulation or aid with adjustments of thetransparency of the OHMD to light reflected from the patient's knee orsurgical theatre.

Someone skilled in the art will readily recognize that all examples andembodiments provided in the foregoing for ACL repair and ACLreconstruction are applicable to all other arthroscopic procedures suchas arthroscopy of the shoulder, hip and ankle and can be applied to manyendoscopic procedures as well as to other embodiments and hipreplacement, knee replacement, spinal surgery, spinal fusion, pediclescrew fixation, vertebroplasty and/or kyphoplasty and many others.

All patents, patent applications, and published references cited hereinare hereby incorporated by reference in their entirety. It should beemphasized that the above-described embodiments of the presentdisclosure are merely possible examples of implementations, merely setforth for a clear understanding of the principles of the disclosure.Many variations and modifications may be made to the above-describedembodiment(s) without departing substantially from the spirit andprinciples of the disclosure. It can be appreciated that several of theabove-disclosed and other features and functions, or alternativesthereof, may be desirably combined into many other different systems orapplications. All such modifications and variations are intended to beincluded herein within the scope of this disclosure, as fall within thescope of the appended claims.

The invention claimed is:
 1. A system for preparing a physical joint ina patient, the system comprising: at least one processor, a see throughoptical head mounted display, a user interface, wherein the system isconfigured to display, by the see through optical head mounted display,at least a portion of a virtual implant component onto a surface of thephysical joint visible directly through the see through optical headmounted display, so as to superimpose the at least portion of thevirtual implant component onto the surface of the physical joint,wherein the at least portion of the virtual implant component is athree-dimensional digital representation corresponding to at least aportion of a physical implant component, a placement indicator of thephysical implant component, or a combination thereof, wherein the systemis configured to superimpose and align the at least portion of thevirtual implant component with at least one anatomic structure of thephysical joint, and wherein the user interface is configured to adjustthe position, orientation or position and orientation of the at leastportion of the virtual implant component in relationship to the at leastone anatomic structure of the physical joint.
 2. The system of claim 1,wherein the system is configured to track the see through optical headmounted display, the virtual implant component, the physical implantcomponent, the physical joint, the at least one anatomic structure, aphysical tool, a physical instrument, a robotic arm or a combinationthereof in a coordinate system.
 3. The system of claim 1, wherein thesystem is configured to maintain the display of the at least portion ofthe virtual implant component in relationship to the at least oneanatomic structure.
 4. The system of claim 3, wherein the system isconfigured to maintain the display of the at least portion of thevirtual implant component in relationship to the at least one anatomicstructure when the joint is moved through a range of motion or when thejoint is moved into different positions corresponding to a flexion,extension, rotation, abduction, adduction or combinations thereof. 5.The system of claim 1, wherein the user interface comprises at least oneof a graphical user interface, a voice recognition, a gesturerecognition, a virtual interface displayed by the see through opticalhead mounted display, a virtual keyboard displayed by the see throughoptical head mounted display, a physical keyboard, a physical computermouse, or a physical track pad.
 6. The system of claim 1, wherein thesystem is for preparing the physical joint of the patient for thephysical implant component, wherein the physical implant component is aknee implant component, a hip implant component, a shoulder jointimplant component, or an ankle joint implant component.
 7. The system ofclaim 1, wherein the user interface is configured to select the virtualimplant component from a library of virtual implant components, whereinthe library of virtual implant components is composed of two or morevirtual implant components of different sizes, different shapes, ordifferent sizes and different shapes, and wherein each virtual implantcomponent of the library is a three-dimensional digital representationcorresponding to at least a portion of a corresponding physical implantcomponent, a placement indicator of a corresponding physical implantcomponent, or a combination thereof.
 8. The system of claim 7, whereinthe user interface is configured to select the virtual implant componentbased on a size, a shape or a size and a shape of the virtual implantcomponent, or wherein the user interface is configured to select thevirtual implant component based on a size, a shape or a size and a shapeof the virtual implant component in relationship to the physical jointof the patient.
 9. The system of claim 1, wherein the user interface isconfigured to adjust a color, brightness, intensity, contrast,transparency or combination thereof of the display of the at leastportion of the virtual implant component by the see through optical headmounted display, and wherein at least a portion of the physical joint isvisible through the superimposed virtual implant component.
 10. Thesystem of claim 7, wherein the system is configured to display two ormore virtual implant components.
 11. The system of claim 10, wherein thesystem is configured to display the two or more virtual implantcomponents with a different color, a different degree of transparency ora combination thereof.
 12. The system of claim 1, wherein the userinterface is configured to move the virtual implant component inrelationship to the physical joint of the patient, in relationship tothe at least one anatomic structure or a combination thereof.
 13. Thesystem of claim 1, wherein the user interface is configured to align thedisplay of the virtual implant component in relationship to the physicaljoint of the patient, in relationship to the at least one anatomicstructure or a combination thereof.
 14. The system of claim 1, whereinthe system is configured to display, by the see through optical headmounted display, the virtual implant component onto the surface of thephysical joint at a predetermined position, predetermined orientation,predetermined rotation, predetermined resection level, predeterminedalignment or combination thereof.
 15. The system of claim 1, wherein thesystem is configured to align the virtual implant component relative toa predetermined rotation axis, a predetermined flexion axis, apredetermined extension axis, a predetermined slope, a predeterminedanteversion, a predetermined inclination or a combination thereof. 16.The system of claim 14, wherein the user interface is configured tomodify the predetermined position, predetermined orientation,predetermined rotation, predetermined resection level, predeterminedalignment or combination thereof of the virtual implant component toadjust for ligamentous laxity or tightness.
 17. The system of claim 14,wherein the predetermined position, predetermined orientation,predetermined rotation, predetermined resection level, predeterminedalignment or combination thereof of the virtual implant component isbased on at least one of a predetermined varus correction, apredetermined valgus correction, a predetermined femoral componentflexion, a predetermined femoral component extension, a predeterminedfemoral component rotation, a predetermined femoral component positionrelative to an anterior cortex, a predetermined tibial component slope,a predetermined tibial component rotation, or a predetermined tibialcomponent position relative to a tibial cortical rim in a kneereplacement, or is based on at least one of a predetermined femoral neckresection for a femoral component, a predetermined leg length, apredetermined femoral component anteversion, a predetermined femoralcomponent offset, a predetermined acetabular component anteversion, or apredetermined acetabular component inclination in a hip replacement. 18.The system of claim 1, wherein the system is configured to display thevirtual implant component at a predetermined position, a predeterminedorientation, a predetermined rotation, a predetermined alignment or acombination thereof relative to an anatomic axis, a biomechanical axis,a deformity or a combination thereof.
 19. The system of claim 1, whereinthe system is configured to display the virtual implant component, bythe see through optical head mounted display, in relationship to thephysical joint of the patient with a predetermined resection level, apredetermined varus angle, a predetermined valgus angle, a predeterminedrotation, a predetermined flexion, a predetermined slope or acombination thereof.
 20. The system of claim 19, wherein the userinterface is configured to modify the position, orientation or positionand orientation of the display of the virtual implant component relativeto the predetermined resection level, predetermined varus angle,predetermined valgus angle, predetermined rotation, predeterminedflexion, predetermined slope or combination thereof.
 21. The system ofclaim 1, wherein the at least one anatomic structure comprises at leastone anatomic landmark, anatomic plane, articular surface, cartilagesurface, subchondral bone surface, cortical bone surface, cut bonesurface, reamed bone surface, milled bone surface, impacted bonesurface, tissue resection, tissue surface, or a combination thereof. 22.The system of claim 1, wherein the at least one anatomic structurecomprises information about an anatomic axis, a biomechanical axis, amechanical axis, a distance between two or more anatomic landmarks, adimensions between two or more anatomic landmarks, a dimension betweenan anterior and a posterior anatomic structure, a dimension between amedial and a lateral anatomic structure, a dimension between a superiorand an inferior anatomic structure, a curvature, a surface, an edge, ashape, a length, a width, a depth, or a combination thereof.
 23. Thesystem of claim 21, wherein the at least one anatomic landmark comprisesat least a portion of or an entire acetabulum, a portion of or an entireedge of an acetabulum, multiple portions of an edge of an acetabulum, aportion of an iliac wall, a portion of a pubic bone, a portion of anischial bone, an anterior superior iliac spine, an anterior inferioriliac spine, a symphysis pubis, a portion of or an entire greatertrochanter, a portion of or an entire lesser trochanter, a center of thefemoral head, a portion of or an entire femoral shaft, a portion of oran entire femoral neck, a portion of or an entire femoral head, a foveacapitis, a transverse acetabular ligament, a pulvinar, a ligamentumteres, a labrum, one or more femoral osteophytes, one or more acetabularosteophytes, a portion of or an entire medial femoral condyle, a portionof or an entire lateral femoral condyle, a portion of or an entirefemoral notch, a portion of or an entire trochlea, a portion of ananterior cortex of the femur, a portion of an anterior cortex of thefemur with adjacent portions of the trochlea, a portion of an anteriorcortex of the femur with adjacent portions of the trochlea andosteophytes, one or more femoral osteophytes, one or more tibialosteophytes, an epicondylar eminence, a medial epicondylar eminence, alateral epicondylar eminence, a portion of or an entire medial tibialplateau, a portion of or an entire lateral tibial plateau, a portion ofor an entire medial tibial spine, a portion of or an entire lateraltibial spine, a portion of an anterior cortex of the tibia, a portion ofan anterior cortex of the tibia and a portion of a tibial plateau, aportion of or an entire patella, a medial edge of a patella, a lateraledge of a patella, a superior pole of a patella, an inferior pole of apatella, a patellar osteophyte, a medial wall of the femoral notch, alateral wall of the femoral notch, a roof of the femoral notch, ananterior cruciate ligament origin, an anterior cruciate ligamentinsertion, a medial wall of the medial condyle, a lateral wall of thelateral condyle, a medial femoral condyle shape, a medial femoralcondyle radius, a medial femoral condyle convexity, a medial femoralcondyle concavity, a lateral femoral condyle shape, a lateral femoralcondyle radius, a lateral femoral condyle convexity, a lateral femoralcondyle concavity, an intercondylar notch shape, an intercondylar notchsurface feature, a medial tibial spine, a lateral tibial spine, ananteromedial tibial rim, an anterolateral tibial rim, a medial tibialrim, a lateral tibial rim, a lowest point of the medial plateau, alowest point of the lateral plateau, a highest point of the medialplateau, a highest point of the lateral plateau, a medial tibial plateaushape, a lateral tibial plateau shape, a medial tibial plateau surface,a medial tibial plateau surface feature, a medial tibial plateau radius,a medial tibial plateau convexity, a medial tibial plateau concavity, alateral tibial plateau shape, a lateral tibial plateau surface, alateral tibial plateau surface feature, a lateral tibial plateau radius,a lateral tibial plateau convexity, a lateral tibial plateau concavity,a portion of or an entire glenoid, a portion of or an entire coracoidprocess, a portion of or an entire acromion, a portion of a clavicle, aportion of or an entire humeral head, a portion of or an entire humeralneck, a portion of a humeral shaft, one or more humeral osteophytes, oneor more glenoid osteophytes, a portion of or an entire glenoid labrum, aportion of or an entire shoulder ligament, a portion of or an entirecoracoacromial ligament, or a portion of or an entire glenohumeralligament.
 24. The system of claim 1, wherein the virtual implantcomponent is a virtual trial femoral component, a virtual trial tibialcomponent, a virtual trial tibial insert, a virtual trial patellarcomponent, a virtual trial acetabular component, a virtual trial glenoidcomponent, or a virtual trial humeral component.
 25. The system of claim1, wherein the joint is a knee and wherein the system is configured todisplay at least a portion of a virtual femoral implant componentsuperimposed onto at least a portion of the femoral articular surface ofthe physical joint and to display at least a portion of a virtual tibialimplant component superimposed onto at least a portion of the tibialarticular surface of the physical joint, wherein the system isconfigured to maintain the display of the at least portion of thevirtual femoral implant component in relationship to the at leastportion of the femoral articular surface and to maintain the display ofthe at least portion of the virtual tibial implant component inrelationship to the at least portion of the tibial articular surfacewhen the knee is moved through a range of motion or when the knee ismoved into different positions of flexion, extension, rotation,abduction, adduction or combinations thereof, or wherein the joint is ahip and wherein the system is configured to display at least a portionof a virtual femoral implant component superimposed onto at least aportion of the proximal femur of the physical joint and to display atleast a portion of a virtual acetabular implant component superimposedonto at least a portion of the acetabular articular surface of thephysical joint, wherein the system is configured to maintain the displayof the at least portion of the virtual femoral implant component inrelationship to the at least portion of the proximal femur and tomaintain the display of the at least portion of the virtual acetabularimplant component in relationship to the at least portion of theacetabular articular surface when the hip is moved through a range ofmotion or when the hip is moved into different positions of flexion,extension, rotation, abduction, adduction or combinations thereof, orwherein the joint is a shoulder and wherein the system is configured todisplay at least a portion of a virtual humeral implant componentsuperimposed onto at least a portion of the proximal humerus of thephysical joint and to display at least a portion of a virtual glenoidimplant component superimposed onto at least a portion of the glenoidarticular surface of the physical joint, wherein the system isconfigured to maintain the display of the at least portion of thevirtual humeral implant component in relationship to the at leastportion of the proximal humerus and to maintain the display of the atleast portion of the virtual glenoid implant component in relationshipto the at least portion of the glenoid articular surface when theshoulder is moved through a range of motion or when the shoulder ismoved into different positions of flexion, extension, rotation,abduction, adduction or combinations thereof.
 26. The system of claim 1,wherein at least one portion of the virtual implant component is tangentor intersecting with the at least one anatomic structure.
 27. The systemof claim 1, wherein the at least one processor is integrated into thesee through optical head mounted display.
 28. The system of claim 1,wherein the at least one processor is separate from the see throughoptical head mounted display.
 29. The system of claim 1, wherein thesystem is configured to use the position, orientation or position andorientation of the virtual implant component in relationship to the atleast one anatomic structure of the physical joint for determining theposition, orientation or position and orientation of at least one boneresection, wherein the at least one bone resection comprises at least abone cut, a drilling, a pinning, a burring, a milling, a reaming, abroaching, an impacting of the physical joint or a combination thereof.30. The system of claim 29, wherein the system further comprises a robotfor the bone resection.